Deadlock in delete_session_cookie with MySQL backend

[trac.edgewall.org@…]

We are running Trac 0.12 using nginx 0.7.65-1ubuntu2 and FastCGI backend (16 Trac instances to balance load across all CPUs, mostly due to hitting #9111). MySQLdb version is 1.2.2-10build1 (Ubuntu). We haven’t had any specific user reports about this yet and I haven’t been able to reproduce it, but there are several logged errors that all look like this:

Backtrace:

2010-10-20 19:11:39,460 Trac[main] ERROR: Internal Server Error:
Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12-py2.6.egg/trac/web/main.py", line 513, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12-py2.6.egg/trac/web/main.py", line 258, in dispatch
    req.session.save()
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12-py2.6.egg/trac/web/session.py", line 88, in save
    @self.env.with_transaction()
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12-py2.6.egg/trac/db/api.py", line 77, in transaction_wrapper
    fn(ldb)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12-py2.6.egg/trac/web/session.py", line 117, in delete_session_cookie
    """, attrs)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12-py2.6.egg/trac/db/util.py", line 85, in executemany
    return self.cursor.executemany(sql_escape_percent(sql), args)
  File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 218, in executemany
    r = self._query('\n'.join([query[:p], ',\n'.join(q), query[e:]]))
  File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 312, in _query
    rowcount = self._do_query(q)
  File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 276, in _do_query
    db.query(q)
OperationalError: (1213, 'Deadlock found when trying to get lock; try restarting transaction')

InnoDB status information:

------------------------
LATEST DETECTED DEADLOCK
------------------------
101020 19:11:39
*** (1) TRANSACTION:
TRANSACTION 0 5521112, ACTIVE 0 sec, process no 16155, OS thread id 47683936683792 inserting
mysql tables in use 1, locked 1
LOCK WAIT 5 lock struct(s), heap size 1216, 3 row lock(s), undo log entries 1
MySQL thread id 4114, query id 14868888 localhost trac_dev update
INSERT INTO session_attribute
                         (sid,authenticated,name,value)
                       VALUES 
('61f6d09bcb39a3a89ed642cb',0,'query_time','1287601898'),
('61f6d09bcb39a3a89ed642cb',0,'query_constraints','[{\'status\': [u\'!closed\'], \'keywords\': [u\'~escape\']}]'),
('61f6d09bcb39a3a89ed642cb',0,'query_href','/query?status=%21closed&keywords=%7Eescape&order=priority'),
('61f6d09bcb39a3a89ed642cb',0,'query_tickets','3435 6093 6429')
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 961 page no 1833 n bits 176 index `PRIMARY` of table `trac_dev`.`session_attribute` trx id 0 5521112 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 76 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166393766373966333266663062376535626565643461; asc 61f97f79f32ff0b7e5beed4a;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 000000440ec0; asc    D  ;; 4: len 7; hex 8000000032016e; asc     2 n;; 5: len 24; hex 363166393766373966333266663062376535626565643461; asc 61f97f79f32ff0b7e5beed4a;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

*** (2) TRANSACTION:
TRANSACTION 0 5521076, ACTIVE 1 sec, process no 16155, OS thread id 47683756410640 fetching rows, thread declared inside InnoDB 305
mysql tables in use 1, locked 1
2180 lock struct(s), heap size 227312, 152433 row lock(s), undo log entries 1
MySQL thread id 4107, query id 14868892 localhost trac_dev updating
DELETE FROM session
                    WHERE authenticated=0 AND last_visit < 1279825898
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 961 page no 1833 n bits 176 index `PRIMARY` of table `trac_dev`.`session_attribute` trx id 0 5521076 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
 0: len 8; hex 73757072656d756d; asc supremum;;

Record lock, heap no 2 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164333262396463376232366131326662363336643664; asc 61d32b9dc7b26a12fb636d6d;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 3: len 6; hex 0000000557f2; asc     W ;; 4: len 7; hex 80000000320167; asc     2 g;; 5: len 24; hex 363164333262396463376232366131326662363336643664; asc 61d32b9dc7b26a12fb636d6d;; 6: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 7: len 30; hex 5b7b27737461747573273a205b752721636c6f736564275d2c2027636f6d; asc [{'status': [u'!closed'], 'com;...(truncated);

Record lock, heap no 3 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164333262396463376232366131326662363336643664; asc 61d32b9dc7b26a12fb636d6d;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f68726566; asc query_href;; 3: len 6; hex 0000000557f2; asc     W ;; 4: len 7; hex 8000000032019f; asc     2  ;; 5: len 24; hex 363164333262396463376232366131326662363336643664; asc 61d32b9dc7b26a12fb636d6d;; 6: len 10; hex 71756572795f68726566; asc query_href;; 7: len 30; hex 2f71756572793f7374617475733d253231636c6f73656426636f6d706f6e; asc /query?status=%21closed&compon;...(truncated);

Record lock, heap no 4 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164333262396463376232366131326662363336643664; asc 61d32b9dc7b26a12fb636d6d;; 1: len 4; hex 80000000; asc     ;; 2: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 3: len 6; hex 0000000557f2; asc     W ;; 4: len 7; hex 800000003201d0; asc     2  ;; 5: len 24; hex 363164333262396463376232366131326662363336643664; asc 61d32b9dc7b26a12fb636d6d;; 6: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 7: len 30; hex 363937312036393739203730373120373134302037313635203731363820; asc 6971 6979 7071 7140 7165 7168 ;...(truncated);

Record lock, heap no 5 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164333262396463376232366131326662363336643664; asc 61d32b9dc7b26a12fb636d6d;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f74696d65; asc query_time;; 3: len 6; hex 0000000557f2; asc     W ;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363164333262396463376232366131326662363336643664; asc 61d32b9dc7b26a12fb636d6d;; 6: len 10; hex 71756572795f74696d65; asc query_time;; 7: len 10; hex 31323836393436383638; asc 1286946868;;

Record lock, heap no 6 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164346263383831616436356463336361316462646237; asc 61d4bc881ad65dc3ca1dbdb7;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 0000004d7380; asc    Ms ;; 4: len 7; hex 8000000032016e; asc     2 n;; 5: len 24; hex 363164346263383831616436356463336361316462646237; asc 61d4bc881ad65dc3ca1dbdb7;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 7 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164346263383831616436356463336361316462646237; asc 61d4bc881ad65dc3ca1dbdb7;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 0000004d7380; asc    Ms ;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363164346263383831616436356463336361316462646237; asc 61d4bc881ad65dc3ca1dbdb7;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 8 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164356366373532326434636463613337633536623335; asc 61d5cf7522d4cdca37c56b35;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 3: len 6; hex 00000015c5b2; asc       ;; 4: len 7; hex 80000000320167; asc     2 g;; 5: len 24; hex 363164356366373532326434636463613337633536623335; asc 61d5cf7522d4cdca37c56b35;; 6: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 7: len 30; hex 5b7b276d696c6573746f6e65273a205b7527312e342e33275d2c20277265; asc [{'milestone': [u'1.4.3'], 're;...(truncated);

Record lock, heap no 9 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164356366373532326434636463613337633536623335; asc 61d5cf7522d4cdca37c56b35;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f68726566; asc query_href;; 3: len 6; hex 00000015c5b2; asc       ;; 4: len 7; hex 8000000032019f; asc     2  ;; 5: len 24; hex 363164356366373532326434636463613337633536623335; asc 61d5cf7522d4cdca37c56b35;; 6: len 10; hex 71756572795f68726566; asc query_href;; 7: len 30; hex 2f71756572793f6d696c6573746f6e653d312e342e33267265706f727465; asc /query?milestone=1.4.3&reporte;...(truncated);

Record lock, heap no 10 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164356366373532326434636463613337633536623335; asc 61d5cf7522d4cdca37c56b35;; 1: len 4; hex 80000000; asc     ;; 2: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 3: len 6; hex 00000015c5b2; asc       ;; 4: len 7; hex 800000003201d0; asc     2  ;; 5: len 24; hex 363164356366373532326434636463613337633536623335; asc 61d5cf7522d4cdca37c56b35;; 6: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 7: len 4; hex 35313233; asc 5123;;

Record lock, heap no 11 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164356366373532326434636463613337633536623335; asc 61d5cf7522d4cdca37c56b35;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f74696d65; asc query_time;; 3: len 6; hex 00000015c5b2; asc       ;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363164356366373532326434636463613337633536623335; asc 61d5cf7522d4cdca37c56b35;; 6: len 10; hex 71756572795f74696d65; asc query_time;; 7: len 10; hex 31323837313536363532; asc 1287156652;;

Record lock, heap no 12 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164366134306166633864656538333033663335336265; asc 61d6a40afc8dee8303f353be;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 0000004dc5cc; asc    M  ;; 4: len 7; hex 800000002d016e; asc     - n;; 5: len 24; hex 363164366134306166633864656538333033663335336265; asc 61d6a40afc8dee8303f353be;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 13 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164366134306166633864656538333033663335336265; asc 61d6a40afc8dee8303f353be;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 0000004dc5cc; asc    M  ;; 4: len 7; hex 800000002d0136; asc     - 6;; 5: len 24; hex 363164366134306166633864656538333033663335336265; asc 61d6a40afc8dee8303f353be;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 14 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164376337366237653862343661363730383635323136; asc 61d7c76b7e8b46a670865216;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 3: len 6; hex 0000001db337; asc      7;; 4: len 7; hex 80000000320167; asc     2 g;; 5: len 24; hex 363164376337366237653862343661363730383635323136; asc 61d7c76b7e8b46a670865216;; 6: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 7: len 30; hex 5b7b27737461747573273a205b752721636c6f736564275d2c20276b6579; asc [{'status': [u'!closed'], 'key;...(truncated);

Record lock, heap no 15 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164376337366237653862343661363730383635323136; asc 61d7c76b7e8b46a670865216;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f68726566; asc query_href;; 3: len 6; hex 0000001db337; asc      7;; 4: len 7; hex 8000000032019f; asc     2  ;; 5: len 24; hex 363164376337366237653862343661363730383635323136; asc 61d7c76b7e8b46a670865216;; 6: len 10; hex 71756572795f68726566; asc query_href;; 7: len 30; hex 2f71756572793f7374617475733d253231636c6f736564266b6579776f72; asc /query?status=%21closed&keywor;...(truncated);

Record lock, heap no 16 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164376337366237653862343661363730383635323136; asc 61d7c76b7e8b46a670865216;; 1: len 4; hex 80000000; asc     ;; 2: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 3: len 6; hex 0000001db337; asc      7;; 4: len 7; hex 800000003201d0; asc     2  ;; 5: len 24; hex 363164376337366237653862343661363730383635323136; asc 61d7c76b7e8b46a670865216;; 6: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 7: len 0; hex ; asc ;;

Record lock, heap no 17 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164376337366237653862343661363730383635323136; asc 61d7c76b7e8b46a670865216;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f74696d65; asc query_time;; 3: len 6; hex 0000001db337; asc      7;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363164376337366237653862343661363730383635323136; asc 61d7c76b7e8b46a670865216;; 6: len 10; hex 71756572795f74696d65; asc query_time;; 7: len 10; hex 31323837323638313534; asc 1287268154;;

Record lock, heap no 18 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164383638653135333964316165323032353232306164; asc 61d868e1539d1ae2025220ad;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 00000042037c; asc    B |;; 4: len 7; hex 8000000032016e; asc     2 n;; 5: len 24; hex 363164383638653135333964316165323032353232306164; asc 61d868e1539d1ae2025220ad;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 19 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164383638653135333964316165323032353232306164; asc 61d868e1539d1ae2025220ad;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 00000042037c; asc    B |;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363164383638653135333964316165323032353232306164; asc 61d868e1539d1ae2025220ad;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 20 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164633164363361396134326665313636613739616331; asc 61dc1d63a9a42fe166a79ac1;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 000000443eb2; asc    D> ;; 4: len 7; hex 8000000032016e; asc     2 n;; 5: len 24; hex 363164633164363361396134326665313636613739616331; asc 61dc1d63a9a42fe166a79ac1;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 21 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164633164363361396134326665313636613739616331; asc 61dc1d63a9a42fe166a79ac1;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 000000443eb2; asc    D> ;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363164633164363361396134326665313636613739616331; asc 61dc1d63a9a42fe166a79ac1;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 22 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164633362383965313139343234663862616366363036; asc 61dc3b89e119424f8bacf606;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 000000248f41; asc    $ A;; 4: len 7; hex 8000000035016e; asc     5 n;; 5: len 24; hex 363164633362383965313139343234663862616366363036; asc 61dc3b89e119424f8bacf606;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 23 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164633362383965313139343234663862616366363036; asc 61dc3b89e119424f8bacf606;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 000000248f41; asc    $ A;; 4: len 7; hex 80000000350136; asc     5 6;; 5: len 24; hex 363164633362383965313139343234663862616366363036; asc 61dc3b89e119424f8bacf606;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 24 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164633865646661326362366534393437616631303336; asc 61dc8edfa2cb6e4947af1036;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 3: len 6; hex 0000004fbcf4; asc    O  ;; 4: len 7; hex 800000002d0167; asc     - g;; 5: len 24; hex 363164633865646661326362366534393437616631303336; asc 61dc8edfa2cb6e4947af1036;; 6: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 7: len 30; hex 5b7b27737461747573273a205b752721636c6f736564275d2c20276b6579; asc [{'status': [u'!closed'], 'key;...(truncated);

Record lock, heap no 25 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164633865646661326362366534393437616631303336; asc 61dc8edfa2cb6e4947af1036;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f68726566; asc query_href;; 3: len 6; hex 0000004fbcf4; asc    O  ;; 4: len 7; hex 800000002d019f; asc     -  ;; 5: len 24; hex 363164633865646661326362366534393437616631303336; asc 61dc8edfa2cb6e4947af1036;; 6: len 10; hex 71756572795f68726566; asc query_href;; 7: len 30; hex 2f71756572793f7374617475733d253231636c6f736564266b6579776f72; asc /query?status=%21closed&keywor;...(truncated);

Record lock, heap no 26 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164633865646661326362366534393437616631303336; asc 61dc8edfa2cb6e4947af1036;; 1: len 4; hex 80000000; asc     ;; 2: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 3: len 6; hex 0000004fbcf4; asc    O  ;; 4: len 7; hex 800000002d01d0; asc     -  ;; 5: len 24; hex 363164633865646661326362366534393437616631303336; asc 61dc8edfa2cb6e4947af1036;; 6: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 7: len 30; hex 333832372036353134203637303520363838302033393330203634343720; asc 3827 6514 6705 6880 3930 6447 ;...(truncated);

Record lock, heap no 27 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164633865646661326362366534393437616631303336; asc 61dc8edfa2cb6e4947af1036;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f74696d65; asc query_time;; 3: len 6; hex 0000004fbcf4; asc    O  ;; 4: len 7; hex 800000002d0136; asc     - 6;; 5: len 24; hex 363164633865646661326362366534393437616631303336; asc 61dc8edfa2cb6e4947af1036;; 6: len 10; hex 71756572795f74696d65; asc query_time;; 7: len 10; hex 31323837353836353538; asc 1287586558;;

Record lock, heap no 28 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164633963396536623835313438323565303738323937; asc 61dc9c9e6b8514825e078297;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 0000003f86eb; asc    ?  ;; 4: len 7; hex 8000000032016e; asc     2 n;; 5: len 24; hex 363164633963396536623835313438323565303738323937; asc 61dc9c9e6b8514825e078297;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 29 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363164633963396536623835313438323565303738323937; asc 61dc9c9e6b8514825e078297;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 0000003f86eb; asc    ?  ;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363164633963396536623835313438323565303738323937; asc 61dc9c9e6b8514825e078297;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 30 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165303830663436656636373936623434383439333964; asc 61e080f46ef6796b4484939d;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 3: len 6; hex 000000494ae2; asc    IJ ;; 4: len 7; hex 800000002d0167; asc     - g;; 5: len 24; hex 363165303830663436656636373936623434383439333964; asc 61e080f46ef6796b4484939d;; 6: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 7: len 30; hex 5b7b27737461747573273a205b752721636c6f736564275d2c20276b6579; asc [{'status': [u'!closed'], 'key;...(truncated);

Record lock, heap no 31 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165303830663436656636373936623434383439333964; asc 61e080f46ef6796b4484939d;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f68726566; asc query_href;; 3: len 6; hex 000000494ae2; asc    IJ ;; 4: len 7; hex 800000002d019f; asc     -  ;; 5: len 24; hex 363165303830663436656636373936623434383439333964; asc 61e080f46ef6796b4484939d;; 6: len 10; hex 71756572795f68726566; asc query_href;; 7: len 30; hex 2f71756572793f7374617475733d253231636c6f736564266b6579776f72; asc /query?status=%21closed&keywor;...(truncated);

Record lock, heap no 32 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165303830663436656636373936623434383439333964; asc 61e080f46ef6796b4484939d;; 1: len 4; hex 80000000; asc     ;; 2: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 3: len 6; hex 000000494ae2; asc    IJ ;; 4: len 7; hex 800000002d01d0; asc     -  ;; 5: len 24; hex 363165303830663436656636373936623434383439333964; asc 61e080f46ef6796b4484939d;; 6: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 7: len 0; hex ; asc ;;

Record lock, heap no 33 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165303830663436656636373936623434383439333964; asc 61e080f46ef6796b4484939d;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f74696d65; asc query_time;; 3: len 6; hex 000000494ae2; asc    IJ ;; 4: len 7; hex 800000002d0136; asc     - 6;; 5: len 24; hex 363165303830663436656636373936623434383439333964; asc 61e080f46ef6796b4484939d;; 6: len 10; hex 71756572795f74696d65; asc query_time;; 7: len 10; hex 31323837353533393535; asc 1287553955;;

Record lock, heap no 34 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165323864646263633166353432626533336162623632; asc 61e28ddbcc1f542be33abb62;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 00000025caec; asc    %  ;; 4: len 7; hex 8000000035016e; asc     5 n;; 5: len 24; hex 363165323864646263633166353432626533336162623632; asc 61e28ddbcc1f542be33abb62;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 35 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165323864646263633166353432626533336162623632; asc 61e28ddbcc1f542be33abb62;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 00000025caec; asc    %  ;; 4: len 7; hex 80000000350136; asc     5 6;; 5: len 24; hex 363165323864646263633166353432626533336162623632; asc 61e28ddbcc1f542be33abb62;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 36 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165333261363664383532656265326334383532386336; asc 61e32a66d852ebe2c48528c6;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 0000003e03dd; asc    >  ;; 4: len 7; hex 8000000032016e; asc     2 n;; 5: len 24; hex 363165333261363664383532656265326334383532386336; asc 61e32a66d852ebe2c48528c6;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 37 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165333261363664383532656265326334383532386336; asc 61e32a66d852ebe2c48528c6;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 0000003e03dd; asc    >  ;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363165333261363664383532656265326334383532386336; asc 61e32a66d852ebe2c48528c6;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 38 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165373865633163666437346631633133353464383639; asc 61e78ec1cfd74f1c1354d869;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 0000002ab261; asc    * a;; 4: len 7; hex 8000000032016e; asc     2 n;; 5: len 24; hex 363165373865633163666437346631633133353464383639; asc 61e78ec1cfd74f1c1354d869;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 39 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165373865633163666437346631633133353464383639; asc 61e78ec1cfd74f1c1354d869;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 0000002ab261; asc    * a;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363165373865633163666437346631633133353464383639; asc 61e78ec1cfd74f1c1354d869;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 40 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165376437363336623130616566326434653237396361; asc 61e7d7636b10aef2d4e279ca;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 0000004a1515; asc    J  ;; 4: len 7; hex 800000002d016e; asc     - n;; 5: len 24; hex 363165376437363336623130616566326434653237396361; asc 61e7d7636b10aef2d4e279ca;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 41 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165376437363336623130616566326434653237396361; asc 61e7d7636b10aef2d4e279ca;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 0000004a1515; asc    J  ;; 4: len 7; hex 800000002d0136; asc     - 6;; 5: len 24; hex 363165376437363336623130616566326434653237396361; asc 61e7d7636b10aef2d4e279ca;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 42 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165393763373963363930396665643537663637333365; asc 61e97c79c6909fed57f6733e;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 3: len 6; hex 00000006d0ae; asc       ;; 4: len 7; hex 80000000320167; asc     2 g;; 5: len 24; hex 363165393763373963363930396665643537663637333365; asc 61e97c79c6909fed57f6733e;; 6: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 7: len 30; hex 5b7b27737461747573273a205b752721636c6f736564275d2c2027747970; asc [{'status': [u'!closed'], 'typ;...(truncated);

Record lock, heap no 43 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165393763373963363930396665643537663637333365; asc 61e97c79c6909fed57f6733e;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f68726566; asc query_href;; 3: len 6; hex 00000006d0ae; asc       ;; 4: len 7; hex 8000000032019f; asc     2  ;; 5: len 24; hex 363165393763373963363930396665643537663637333365; asc 61e97c79c6909fed57f6733e;; 6: len 10; hex 71756572795f68726566; asc query_href;; 7: len 30; hex 2f71756572793f7374617475733d253231636c6f73656426747970653d62; asc /query?status=%21closed&type=b;...(truncated);

Record lock, heap no 44 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165393763373963363930396665643537663637333365; asc 61e97c79c6909fed57f6733e;; 1: len 4; hex 80000000; asc     ;; 2: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 3: len 6; hex 00000006d0ae; asc       ;; 4: len 7; hex 800000003201d0; asc     2  ;; 5: len 24; hex 363165393763373963363930396665643537663637333365; asc 61e97c79c6909fed57f6733e;; 6: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 7: len 30; hex 353734312035373932203538383620363430302036393733203730313920; asc 5741 5792 5886 6400 6973 7019 ;...(truncated);

Record lock, heap no 45 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165393763373963363930396665643537663637333365; asc 61e97c79c6909fed57f6733e;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f74696d65; asc query_time;; 3: len 6; hex 00000006d0ae; asc       ;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363165393763373963363930396665643537663637333365; asc 61e97c79c6909fed57f6733e;; 6: len 10; hex 71756572795f74696d65; asc query_time;; 7: len 10; hex 31323836393638303833; asc 1286968083;;

Record lock, heap no 46 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165626432343537663166633861316565323831663236; asc 61ebd2457f1fc8a1ee281f26;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 3: len 6; hex 000000114fbb; asc     O ;; 4: len 7; hex 80000000320167; asc     2 g;; 5: len 24; hex 363165626432343537663166633861316565323831663236; asc 61ebd2457f1fc8a1ee281f26;; 6: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 7: len 30; hex 5b7b27737461747573273a205b752721636c6f736564275d2c20276b6579; asc [{'status': [u'!closed'], 'key;...(truncated);

Record lock, heap no 47 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165626432343537663166633861316565323831663236; asc 61ebd2457f1fc8a1ee281f26;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f68726566; asc query_href;; 3: len 6; hex 000000114fbb; asc     O ;; 4: len 7; hex 8000000032019f; asc     2  ;; 5: len 24; hex 363165626432343537663166633861316565323831663236; asc 61ebd2457f1fc8a1ee281f26;; 6: len 10; hex 71756572795f68726566; asc query_href;; 7: len 30; hex 2f71756572793f7374617475733d253231636c6f736564266b6579776f72; asc /query?status=%21closed&keywor;...(truncated);

Record lock, heap no 48 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165626432343537663166633861316565323831663236; asc 61ebd2457f1fc8a1ee281f26;; 1: len 4; hex 80000000; asc     ;; 2: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 3: len 6; hex 000000114fbb; asc     O ;; 4: len 7; hex 800000003201d0; asc     2  ;; 5: len 24; hex 363165626432343537663166633861316565323831663236; asc 61ebd2457f1fc8a1ee281f26;; 6: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 7: len 0; hex ; asc ;;

Record lock, heap no 49 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165626432343537663166633861316565323831663236; asc 61ebd2457f1fc8a1ee281f26;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f74696d65; asc query_time;; 3: len 6; hex 000000114fbb; asc     O ;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363165626432343537663166633861316565323831663236; asc 61ebd2457f1fc8a1ee281f26;; 6: len 10; hex 71756572795f74696d65; asc query_time;; 7: len 10; hex 31323837303935373531; asc 1287095751;;

Record lock, heap no 50 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165636435663534613062663162356162346536646535; asc 61ecd5f54a0bf1b5ab4e6de5;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 3: len 6; hex 00000032d57e; asc    2 ~;; 4: len 7; hex 80000000350167; asc     5 g;; 5: len 24; hex 363165636435663534613062663162356162346536646535; asc 61ecd5f54a0bf1b5ab4e6de5;; 6: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 7: len 30; hex 5b7b27737461747573273a205b752721636c6f736564275d2c2027726570; asc [{'status': [u'!closed'], 'rep;...(truncated);

Record lock, heap no 51 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165636435663534613062663162356162346536646535; asc 61ecd5f54a0bf1b5ab4e6de5;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f68726566; asc query_href;; 3: len 6; hex 00000032d57e; asc    2 ~;; 4: len 7; hex 8000000035019f; asc     5  ;; 5: len 24; hex 363165636435663534613062663162356162346536646535; asc 61ecd5f54a0bf1b5ab4e6de5;; 6: len 10; hex 71756572795f68726566; asc query_href;; 7: len 30; hex 2f71756572793f7374617475733d253231636c6f736564267265706f7274; asc /query?status=%21closed&report;...(truncated);

Record lock, heap no 52 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165636435663534613062663162356162346536646535; asc 61ecd5f54a0bf1b5ab4e6de5;; 1: len 4; hex 80000000; asc     ;; 2: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 3: len 6; hex 00000032d57e; asc    2 ~;; 4: len 7; hex 800000003501d0; asc     5  ;; 5: len 24; hex 363165636435663534613062663162356162346536646535; asc 61ecd5f54a0bf1b5ab4e6de5;; 6: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 7: len 0; hex ; asc ;;

Record lock, heap no 53 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165636435663534613062663162356162346536646535; asc 61ecd5f54a0bf1b5ab4e6de5;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f74696d65; asc query_time;; 3: len 6; hex 00000032d57e; asc    2 ~;; 4: len 7; hex 80000000350136; asc     5 6;; 5: len 24; hex 363165636435663534613062663162356162346536646535; asc 61ecd5f54a0bf1b5ab4e6de5;; 6: len 10; hex 71756572795f74696d65; asc query_time;; 7: len 10; hex 31323837343330383232; asc 1287430822;;

Record lock, heap no 54 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165643932303363663938633736623839373438646335; asc 61ed9203cf98c76b89748dc5;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 00000024087c; asc    $ |;; 4: len 7; hex 8000000032016e; asc     2 n;; 5: len 24; hex 363165643932303363663938633736623839373438646335; asc 61ed9203cf98c76b89748dc5;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 55 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165643932303363663938633736623839373438646335; asc 61ed9203cf98c76b89748dc5;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 00000024087c; asc    $ |;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363165643932303363663938633736623839373438646335; asc 61ed9203cf98c76b89748dc5;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 56 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165653236343334363038323736336564646631323764; asc 61ee264346082763eddf127d;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 0000002a7707; asc    *w ;; 4: len 7; hex 8000000032016e; asc     2 n;; 5: len 24; hex 363165653236343334363038323736336564646631323764; asc 61ee264346082763eddf127d;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 57 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165653236343334363038323736336564646631323764; asc 61ee264346082763eddf127d;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 0000002a7707; asc    *w ;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363165653236343334363038323736336564646631323764; asc 61ee264346082763eddf127d;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 58 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165663866393937383439646466303661636437326332; asc 61ef8f997849ddf06acd72c2;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 0000004cf0a3; asc    L  ;; 4: len 7; hex 800000002d016e; asc     - n;; 5: len 24; hex 363165663866393937383439646466303661636437326332; asc 61ef8f997849ddf06acd72c2;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 59 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165663866393937383439646466303661636437326332; asc 61ef8f997849ddf06acd72c2;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 0000004cf0a3; asc    L  ;; 4: len 7; hex 800000002d0136; asc     - 6;; 5: len 24; hex 363165663866393937383439646466303661636437326332; asc 61ef8f997849ddf06acd72c2;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 60 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165666330666462366338376262333436626664336433; asc 61efc0fdb6c87bb346bfd3d3;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 3: len 6; hex 0000002c9d7e; asc    , ~;; 4: len 7; hex 80000000350167; asc     5 g;; 5: len 24; hex 363165666330666462366338376262333436626664336433; asc 61efc0fdb6c87bb346bfd3d3;; 6: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 7: len 23; hex 5b7b27737461747573273a205b75276f70656e275d7d5d; asc [{'status': [u'open']}];;

Record lock, heap no 61 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165666330666462366338376262333436626664336433; asc 61efc0fdb6c87bb346bfd3d3;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f68726566; asc query_href;; 3: len 6; hex 0000002c9d7e; asc    , ~;; 4: len 7; hex 8000000035019f; asc     5  ;; 5: len 24; hex 363165666330666462366338376262333436626664336433; asc 61efc0fdb6c87bb346bfd3d3;; 6: len 10; hex 71756572795f68726566; asc query_href;; 7: len 30; hex 2f71756572793f7374617475733d6f70656e267265706f72743d3438266f; asc /query?status=open&report=48&o;...(truncated);

Record lock, heap no 62 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165666330666462366338376262333436626664336433; asc 61efc0fdb6c87bb346bfd3d3;; 1: len 4; hex 80000000; asc     ;; 2: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 3: len 6; hex 0000002c9d7e; asc    , ~;; 4: len 7; hex 800000003501d0; asc     5  ;; 5: len 24; hex 363165666330666462366338376262333436626664336433; asc 61efc0fdb6c87bb346bfd3d3;; 6: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 7: len 30; hex 323435372033303136203632343220363331392036343230203634343620; asc 2457 3016 6242 6319 6420 6446 ;...(truncated);

Record lock, heap no 63 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363165666330666462366338376262333436626664336433; asc 61efc0fdb6c87bb346bfd3d3;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f74696d65; asc query_time;; 3: len 6; hex 0000002c9d7e; asc    , ~;; 4: len 7; hex 80000000350136; asc     5 6;; 5: len 24; hex 363165666330666462366338376262333436626664336433; asc 61efc0fdb6c87bb346bfd3d3;; 6: len 10; hex 71756572795f74696d65; asc query_time;; 7: len 10; hex 31323837333939393637; asc 1287399967;;

Record lock, heap no 64 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166303439383464643633316333353637343134316664; asc 61f04984dd631c35674141fd;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 3: len 6; hex 0000004f3ff5; asc    O? ;; 4: len 7; hex 800000002d0167; asc     - g;; 5: len 24; hex 363166303439383464643633316333353637343134316664; asc 61f04984dd631c35674141fd;; 6: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 7: len 30; hex 5b7b27737461747573273a205b752721636c6f736564275d2c20276b6579; asc [{'status': [u'!closed'], 'key;...(truncated);

Record lock, heap no 65 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166303439383464643633316333353637343134316664; asc 61f04984dd631c35674141fd;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f68726566; asc query_href;; 3: len 6; hex 0000004f3ff5; asc    O? ;; 4: len 7; hex 800000002d019f; asc     -  ;; 5: len 24; hex 363166303439383464643633316333353637343134316664; asc 61f04984dd631c35674141fd;; 6: len 10; hex 71756572795f68726566; asc query_href;; 7: len 30; hex 2f71756572793f7374617475733d253231636c6f736564266b6579776f72; asc /query?status=%21closed&keywor;...(truncated);

Record lock, heap no 66 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166303439383464643633316333353637343134316664; asc 61f04984dd631c35674141fd;; 1: len 4; hex 80000000; asc     ;; 2: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 3: len 6; hex 0000004f3ff5; asc    O? ;; 4: len 7; hex 800000002d01d0; asc     -  ;; 5: len 24; hex 363166303439383464643633316333353637343134316664; asc 61f04984dd631c35674141fd;; 6: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 7: len 30; hex 373231322037323139203732323020373232352037323433203732343720; asc 7212 7219 7220 7225 7243 7247 ;...(truncated);

Record lock, heap no 67 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166303439383464643633316333353637343134316664; asc 61f04984dd631c35674141fd;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f74696d65; asc query_time;; 3: len 6; hex 0000004f3ff5; asc    O? ;; 4: len 7; hex 800000002d0136; asc     - 6;; 5: len 24; hex 363166303439383464643633316333353637343134316664; asc 61f04984dd631c35674141fd;; 6: len 10; hex 71756572795f74696d65; asc query_time;; 7: len 10; hex 31323837353832383030; asc 1287582800;;

Record lock, heap no 68 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166306161336261393562373331333236396137393539; asc 61f0aa3ba95b7313269a7959;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 000000291d1b; asc    )  ;; 4: len 7; hex 8000000032016e; asc     2 n;; 5: len 24; hex 363166306161336261393562373331333236396137393539; asc 61f0aa3ba95b7313269a7959;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 69 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166306161336261393562373331333236396137393539; asc 61f0aa3ba95b7313269a7959;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 000000291d1b; asc    )  ;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363166306161336261393562373331333236396137393539; asc 61f0aa3ba95b7313269a7959;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 70 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166323231656337626333633330663261303864316330; asc 61f221ec7bc3c30f2a08d1c0;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 3: len 6; hex 00000043da0c; asc    C  ;; 4: len 7; hex 800000002d0167; asc     - g;; 5: len 24; hex 363166323231656337626333633330663261303864316330; asc 61f221ec7bc3c30f2a08d1c0;; 6: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 7: len 30; hex 5b7b27737461747573273a205b752721636c6f736564275d2c2027726570; asc [{'status': [u'!closed'], 'rep;...(truncated);

Record lock, heap no 71 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166323231656337626333633330663261303864316330; asc 61f221ec7bc3c30f2a08d1c0;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f68726566; asc query_href;; 3: len 6; hex 00000043da0c; asc    C  ;; 4: len 7; hex 800000002d019f; asc     -  ;; 5: len 24; hex 363166323231656337626333633330663261303864316330; asc 61f221ec7bc3c30f2a08d1c0;; 6: len 10; hex 71756572795f68726566; asc query_href;; 7: len 30; hex 2f71756572793f7374617475733d253231636c6f736564267265706f7274; asc /query?status=%21closed&report;...(truncated);

Record lock, heap no 72 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166323231656337626333633330663261303864316330; asc 61f221ec7bc3c30f2a08d1c0;; 1: len 4; hex 80000000; asc     ;; 2: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 3: len 6; hex 00000043da0c; asc    C  ;; 4: len 7; hex 800000002d01d0; asc     -  ;; 5: len 24; hex 363166323231656337626333633330663261303864316330; asc 61f221ec7bc3c30f2a08d1c0;; 6: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 7: len 0; hex ; asc ;;

Record lock, heap no 73 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166323231656337626333633330663261303864316330; asc 61f221ec7bc3c30f2a08d1c0;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f74696d65; asc query_time;; 3: len 6; hex 00000043da0c; asc    C  ;; 4: len 7; hex 800000002d0136; asc     - 6;; 5: len 24; hex 363166323231656337626333633330663261303864316330; asc 61f221ec7bc3c30f2a08d1c0;; 6: len 10; hex 71756572795f74696d65; asc query_time;; 7: len 10; hex 31323837353332303337; asc 1287532037;;

Record lock, heap no 74 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166323536613361613631373663333435343535623264; asc 61f256a3aa6176c345455b2d;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 0000000a7f5c; asc      \;; 4: len 7; hex 8000000032016e; asc     2 n;; 5: len 24; hex 363166323536613361613631373663333435343535623264; asc 61f256a3aa6176c345455b2d;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 75 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166323536613361613631373663333435343535623264; asc 61f256a3aa6176c345455b2d;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 0000000a7f5c; asc      \;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363166323536613361613631373663333435343535623264; asc 61f256a3aa6176c345455b2d;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 76 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166393766373966333266663062376535626565643461; asc 61f97f79f32ff0b7e5beed4a;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 000000440ec0; asc    D  ;; 4: len 7; hex 8000000032016e; asc     2 n;; 5: len 24; hex 363166393766373966333266663062376535626565643461; asc 61f97f79f32ff0b7e5beed4a;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 77 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166393766373966333266663062376535626565643461; asc 61f97f79f32ff0b7e5beed4a;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 000000440ec0; asc    D  ;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363166393766373966333266663062376535626565643461; asc 61f97f79f32ff0b7e5beed4a;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 78 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166623663356161326364336538626665646536656361; asc 61fb6c5aa2cd3e8bfede6eca;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 0000004738bf; asc    G8 ;; 4: len 7; hex 8000000032016e; asc     2 n;; 5: len 24; hex 363166623663356161326364336538626665646536656361; asc 61fb6c5aa2cd3e8bfede6eca;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 79 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166623663356161326364336538626665646536656361; asc 61fb6c5aa2cd3e8bfede6eca;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 0000004738bf; asc    G8 ;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363166623663356161326364336538626665646536656361; asc 61fb6c5aa2cd3e8bfede6eca;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 80 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166633264626139373431616135643064346333616663; asc 61fc2dba9741aa5d0d4c3afc;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 3: len 6; hex 00000041960c; asc    A  ;; 4: len 7; hex 80000000320167; asc     2 g;; 5: len 24; hex 363166633264626139373431616135643064346333616663; asc 61fc2dba9741aa5d0d4c3afc;; 6: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 7: len 30; hex 5b7b27737461747573273a205b752721636c6f736564275d2c2027726570; asc [{'status': [u'!closed'], 'rep;...(truncated);

Record lock, heap no 81 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166633264626139373431616135643064346333616663; asc 61fc2dba9741aa5d0d4c3afc;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f68726566; asc query_href;; 3: len 6; hex 00000041960c; asc    A  ;; 4: len 7; hex 8000000032019f; asc     2  ;; 5: len 24; hex 363166633264626139373431616135643064346333616663; asc 61fc2dba9741aa5d0d4c3afc;; 6: len 10; hex 71756572795f68726566; asc query_href;; 7: len 30; hex 2f71756572793f7374617475733d253231636c6f736564267265706f7274; asc /query?status=%21closed&report;...(truncated);

Record lock, heap no 82 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166633264626139373431616135643064346333616663; asc 61fc2dba9741aa5d0d4c3afc;; 1: len 4; hex 80000000; asc     ;; 2: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 3: len 6; hex 00000041960c; asc    A  ;; 4: len 7; hex 800000003201d0; asc     2  ;; 5: len 24; hex 363166633264626139373431616135643064346333616663; asc 61fc2dba9741aa5d0d4c3afc;; 6: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 7: len 0; hex ; asc ;;

Record lock, heap no 83 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166633264626139373431616135643064346333616663; asc 61fc2dba9741aa5d0d4c3afc;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f74696d65; asc query_time;; 3: len 6; hex 00000041960c; asc    A  ;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363166633264626139373431616135643064346333616663; asc 61fc2dba9741aa5d0d4c3afc;; 6: len 10; hex 71756572795f74696d65; asc query_time;; 7: len 10; hex 31323837353232383535; asc 1287522855;;

Record lock, heap no 84 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166633365376663646232623361646239616239643265; asc 61fc3e7fcdb2b3adb9ab9d2e;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 0000001bbe74; asc      t;; 4: len 7; hex 8000000032016e; asc     2 n;; 5: len 24; hex 363166633365376663646232623361646239616239643265; asc 61fc3e7fcdb2b3adb9ab9d2e;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 85 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166633365376663646232623361646239616239643265; asc 61fc3e7fcdb2b3adb9ab9d2e;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 0000001bbe74; asc      t;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363166633365376663646232623361646239616239643265; asc 61fc3e7fcdb2b3adb9ab9d2e;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 86 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166643232633134623165333839383237336362643530; asc 61fd22c14b1e3898273cbd50;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 3: len 6; hex 0000004fc67e; asc    O ~;; 4: len 7; hex 800000002d0167; asc     - g;; 5: len 24; hex 363166643232633134623165333839383237336362643530; asc 61fd22c14b1e3898273cbd50;; 6: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 7: len 30; hex 5b7b27737461747573273a205b752721636c6f736564275d2c2027726570; asc [{'status': [u'!closed'], 'rep;...(truncated);

Record lock, heap no 87 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166643232633134623165333839383237336362643530; asc 61fd22c14b1e3898273cbd50;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f68726566; asc query_href;; 3: len 6; hex 0000004fc67e; asc    O ~;; 4: len 7; hex 800000002d019f; asc     -  ;; 5: len 24; hex 363166643232633134623165333839383237336362643530; asc 61fd22c14b1e3898273cbd50;; 6: len 10; hex 71756572795f68726566; asc query_href;; 7: len 30; hex 2f71756572793f7374617475733d253231636c6f736564267265706f7274; asc /query?status=%21closed&report;...(truncated);

Record lock, heap no 88 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166643232633134623165333839383237336362643530; asc 61fd22c14b1e3898273cbd50;; 1: len 4; hex 80000000; asc     ;; 2: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 3: len 6; hex 0000004fc67e; asc    O ~;; 4: len 7; hex 800000002d01d0; asc     -  ;; 5: len 24; hex 363166643232633134623165333839383237336362643530; asc 61fd22c14b1e3898273cbd50;; 6: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 7: len 0; hex ; asc ;;

Record lock, heap no 89 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166643232633134623165333839383237336362643530; asc 61fd22c14b1e3898273cbd50;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f74696d65; asc query_time;; 3: len 6; hex 0000004fc67e; asc    O ~;; 4: len 7; hex 800000002d0136; asc     - 6;; 5: len 24; hex 363166643232633134623165333839383237336362643530; asc 61fd22c14b1e3898273cbd50;; 6: len 10; hex 71756572795f74696d65; asc query_time;; 7: len 10; hex 31323837353836363335; asc 1287586635;;

Record lock, heap no 90 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166653333633638373739363334316137303365366335; asc 61fe33c687796341a703e6c5;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 00000041014c; asc    A L;; 4: len 7; hex 8000000032016e; asc     2 n;; 5: len 24; hex 363166653333633638373739363334316137303365366335; asc 61fe33c687796341a703e6c5;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 91 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363166653333633638373739363334316137303365366335; asc 61fe33c687796341a703e6c5;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 00000041014c; asc    A L;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363166653333633638373739363334316137303365366335; asc 61fe33c687796341a703e6c5;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 92 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363230306132366237353930356537313964303766376134; asc 6200a26b75905e719d07f7a4;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 0000002d3a2d; asc    -:-;; 4: len 7; hex 8000000035016e; asc     5 n;; 5: len 24; hex 363230306132366237353930356537313964303766376134; asc 6200a26b75905e719d07f7a4;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 93 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363230306132366237353930356537313964303766376134; asc 6200a26b75905e719d07f7a4;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 0000002d3a2d; asc    -:-;; 4: len 7; hex 80000000350136; asc     5 6;; 5: len 24; hex 363230306132366237353930356537313964303766376134; asc 6200a26b75905e719d07f7a4;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 94 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363230333837363362363430643039653436386263343264; asc 62038763b640d09e468bc42d;; 1: len 4; hex 80000000; asc     ;; 2: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 3: len 6; hex 00000011baab; asc       ;; 4: len 7; hex 8000000032016e; asc     2 n;; 5: len 24; hex 363230333837363362363430643039653436386263343264; asc 62038763b640d09e468bc42d;; 6: len 16; hex 74696d656c696e652e617574686f7273; asc timeline.authors;; 7: len 0; hex ; asc ;;

Record lock, heap no 95 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363230333837363362363430643039653436386263343264; asc 62038763b640d09e468bc42d;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 3: len 6; hex 00000011baab; asc       ;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363230333837363362363430643039653436386263343264; asc 62038763b640d09e468bc42d;; 6: len 17; hex 74696d656c696e652e646179736261636b; asc timeline.daysback;; 7: len 1; hex 37; asc 7;;

Record lock, heap no 96 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363230336364393634303636306430613936656539336230; asc 6203cd9640660d0a96ee93b0;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 3: len 6; hex 0000000dd3b2; asc       ;; 4: len 7; hex 80000000320167; asc     2 g;; 5: len 24; hex 363230336364393634303636306430613936656539336230; asc 6203cd9640660d0a96ee93b0;; 6: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 7: len 30; hex 5b7b27737461747573273a205b752721636c6f736564275d2c20276b6579; asc [{'status': [u'!closed'], 'key;...(truncated);

Record lock, heap no 97 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363230336364393634303636306430613936656539336230; asc 6203cd9640660d0a96ee93b0;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f68726566; asc query_href;; 3: len 6; hex 0000000dd3b2; asc       ;; 4: len 7; hex 8000000032019f; asc     2  ;; 5: len 24; hex 363230336364393634303636306430613936656539336230; asc 6203cd9640660d0a96ee93b0;; 6: len 10; hex 71756572795f68726566; asc query_href;; 7: len 30; hex 2f71756572793f7374617475733d253231636c6f736564266b6579776f72; asc /query?status=%21closed&keywor;...(truncated);

Record lock, heap no 98 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363230336364393634303636306430613936656539336230; asc 6203cd9640660d0a96ee93b0;; 1: len 4; hex 80000000; asc     ;; 2: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 3: len 6; hex 0000000dd3b2; asc       ;; 4: len 7; hex 800000003201d0; asc     2  ;; 5: len 24; hex 363230336364393634303636306430613936656539336230; asc 6203cd9640660d0a96ee93b0;; 6: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 7: len 14; hex 3430383820343537362036343939; asc 4088 4576 6499;;

Record lock, heap no 99 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363230336364393634303636306430613936656539336230; asc 6203cd9640660d0a96ee93b0;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f74696d65; asc query_time;; 3: len 6; hex 0000000dd3b2; asc       ;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363230336364393634303636306430613936656539336230; asc 6203cd9640660d0a96ee93b0;; 6: len 10; hex 71756572795f74696d65; asc query_time;; 7: len 10; hex 31323837303636353433; asc 1287066543;;

Record lock, heap no 100 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363230376138643465643062383166336332653462393436; asc 6207a8d4ed0b81f3c2e4b946;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f68726566; asc query_href;; 3: len 6; hex 0000001b0af5; asc       ;; 4: len 7; hex 80000000320136; asc     2 6;; 5: len 24; hex 363230376138643465643062383166336332653462393436; asc 6207a8d4ed0b81f3c2e4b946;; 6: len 10; hex 71756572795f68726566; asc query_href;; 7: len 30; hex 2f7265706f72742f34343f736f72743d6d696c6573746f6e65266173633d; asc /report/44?sort=milestone&asc=;...(truncated);

Record lock, heap no 101 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363230376138643465643062383166336332653462393436; asc 6207a8d4ed0b81f3c2e4b946;; 1: len 4; hex 80000000; asc     ;; 2: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 3: len 6; hex 0000001b0af5; asc       ;; 4: len 7; hex 80000000320167; asc     2 g;; 5: len 24; hex 363230376138643465643062383166336332653462393436; asc 6207a8d4ed0b81f3c2e4b946;; 6: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 7: len 30; hex 373036352037303130203730363120373136392036343938203632333820; asc 7065 7010 7061 7169 6498 6238 ;...(truncated);

Record lock, heap no 102 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363230386365333036333630363036306264646666386365; asc 6208ce3063606060bddff8ce;; 1: len 4; hex 80000000; asc     ;; 2: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 3: len 6; hex 00000046245e; asc    F$^;; 4: len 7; hex 80000000320167; asc     2 g;; 5: len 24; hex 363230386365333036333630363036306264646666386365; asc 6208ce3063606060bddff8ce;; 6: len 17; hex 71756572795f636f6e73747261696e7473; asc query_constraints;; 7: len 30; hex 5b7b27737461747573273a205b752721636c6f736564275d2c2027726570; asc [{'status': [u'!closed'], 'rep;...(truncated);

Record lock, heap no 103 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363230386365333036333630363036306264646666386365; asc 6208ce3063606060bddff8ce;; 1: len 4; hex 80000000; asc     ;; 2: len 10; hex 71756572795f68726566; asc query_href;; 3: len 6; hex 00000046245e; asc    F$^;; 4: len 7; hex 8000000032019f; asc     2  ;; 5: len 24; hex 363230386365333036333630363036306264646666386365; asc 6208ce3063606060bddff8ce;; 6: len 10; hex 71756572795f68726566; asc query_href;; 7: len 30; hex 2f71756572793f7374617475733d253231636c6f736564267265706f7274; asc /query?status=%21closed&report;...(truncated);

Record lock, heap no 104 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 363230386365333036333630363036306264646666386365; asc 6208ce3063606060bddff8ce;; 1: len 4; hex 80000000; asc     ;; 2: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 3: len 6; hex 00000046245e; asc    F$^;; 4: len 7; hex 800000003201d0; asc     2  ;; 5: len 24; hex 363230386365333036333630363036306264646666386365; asc 6208ce3063606060bddff8ce;; 6: len 13; hex 71756572795f7469636b657473; asc query_tickets;; 7: len 4; hex 36363338; asc 6638;;

Record lock, heap no 105 PHYSICAL RECORD: n_fields 8; compact format; info bits 0
 0: len 24; hex 3632303863653330363336303

Let me know if more information is needed.

Regards,

[Christian Boos]

You know that #9111 is supposed to be fixed by now? Please upgrade to 0.12.1 ;-)

For the specific problem about the lock, it seems to be a MySQL issue rather than a Trac issue. In the About page, when logged with the TRAC_ADMIN permission, you'll see some important details about the MySqlDB bindings, notably if they're threadsafe or not.

[OP]

MySQLdb is reportedly thread safe: server: "5.1.41-3ubuntu12.6", client: "5.1.41", thread-safe: 1

Not sure what else I can provide here…

[OP]

Updated to 0.12.1, and this problem persists. Actually, it seems that it may even be worse now.

From my understanding of ​InnoDB deadlocks, it’s Trac’s responsibility to gracefully handle them, by retrying transactions, by using smaller transactions, or by performing more aggressive locking if certain operations can’t reasonably occur concurrently with other operations.

In this case, I’d suggest starting with splitting delete_session_cookie into smaller transactions; the stuff inserting/updating the current session is irrelevant to the stuff that purges old sessions, so make them into two separate transactions.

If all else fails, performing a table lock or setting the isolation mode to serializable should probably take care of it at the expense of concurrency—but SQLite doesn’t support explicit table locking (PG and MySQL do).

[Christian Boos]

Good points, thanks for the suggestions. I think we should be more careful with transactions affecting multiple tables, that doesn't work well in general.

[Christian Boos]

Mr. zetafleet.com, please try out the following patch and tell us if the locks are still there after that.

Note that with the patch, one unit test fails (test_update_session (trac.web.tests.session.SessionTestCase). This is because while reworking the Session.save() code, I observed the Purge expired sessions. We do this only when the session was changed as to minimize the purging comment. But that logic, introduced in r1215, was actually broken by r3256, and the unit test relies on the new behavior. I'd rather revert to the original idea and fix the test. Opinions?

[Christian Boos]

session: try to avoid table deadlocks

[Christian Boos]

Ok, I think this needs more work; as locks are done at the row level and not at the table level, it doesn't really matter if we take care of accessing always the session table before the session_attribute table, what actually matters is the behavior at the level of the rows, which makes the matter even more complicated.

Updated patch follows. It is probably not the last word on this, but it would nevertheless be nice to have some feedback to see if there's any improvement.

[Christian Boos]

session: split purge transaction in two (applies on 9705-split-save-purge.diff​)

[OP]

I will get this installed on Monday and report back how it is working.

[OP]

Sorry for the trouble, but could you possibly provide a patchfile that will apply against 0.12.1? I’m not able to apply these patches (all hunks fail) and I can’t deploy a WIP version of Trac. Thanks!

[Christian Boos]

combined patch rebased on 0.12.1

[Christian Boos]

combined patch (on r10420)

[Christian Boos]

See 9705-mysql-session-save--trac-0.12.1.patch​.

Now also testing the 0.12.2dev version of the patch on edgewall.org's 0.12 installations.

[anonymous]

OK, OP here. After a bunch of struggling (WHY did easy_install install Trac with DOS-style line endings?!), I finally got the patch applied to our installation, so I will make sure it is monitored over the week and let you know if we still see any deadlocks.

Thanks!

[Christian Boos]

Replying to anonymous:

(WHY did easy_install install Trac with DOS-style line endings?!)

For some reason (ordering of entries on PyPI?), easy_install picks either the .tar.gz or the .zip file. The latter contains files in DOS-line ending, unfortunately.

[Christian Boos]

I'm going to apply the patch today. It would be nice to get some feedback before the actual release though, at least based on the 0.12.2rc1 version which we're going to produce later today.

[Thijs Triemstra]

Replying to cboos:

the 0.12.2rc1 version which we're going to produce later today.

Woo! :)

[Christian Boos]

Both patches applied, in r10459 and r10460.

Please reopen if the problem persists with 0.12.2rc1 (still due "today", i.e. very soon now but as time permits… ;-) ).

[OP]

New problem.

2011-01-23 00:56:05,844 Trac[main] ERROR: Internal Server Error:
Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.1-py2.6.egg/trac/web/main.py", line 511, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.1-py2.6.egg/trac/web/main.py", line 260, in dispatch
    req.session.save()
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.1-py2.6.egg/trac/web/session.py", line 102, in save
    @self.env.with_transaction()
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.1-py2.6.egg/trac/db/api.py", line 77, in transaction_wrapper
    fn(ldb)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.1-py2.6.egg/trac/web/session.py", line 143, in delete_session_cookie
    """, [(self.sid, authenticated, k, v) for k, v in items])
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.1-py2.6.egg/trac/db/util.py", line 85, in executemany
    return self.cursor.executemany(sql_escape_percent(sql), args)
  File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 218, in executemany
    r = self._query('\n'.join([query[:p], ',\n'.join(q), query[e:]]))
  File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 312, in _query
    rowcount = self._do_query(q)
  File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 276, in _do_query
    db.query(q)
OperationalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')

[Christian Boos]

Ok, and what about the frequency of issues? Has the change somewhat improved the situation or instead worsened it?

[OP]

Frequency seems to be reduced, though I have also perceived a sharp decrease in page load performance (versus commenting out the entire session garbage collection feature and replacing it with a nightly cron task).

[Christian Boos]

Replying to OP:

Frequency seems to be reduced, though I have also perceived a sharp decrease in page load performance (versus commenting out the entire session garbage collection feature and replacing it with a nightly cron task).

Oh, interesting to know… I suppose you introduced the nightly cron after seeing the original table lock issues, or did you have those locks anyway?

The purge frequency should already be once a day (for a given user), but maybe we could reduce that even more (once a day globally). So the performance impact should normally not affect every page load. What do you see exactly? A performance impact on every page load? When you change something in your session data (executing a new custom query, saving diff options, timeline options, other user preferences…)?

Also, can you give us an idea about the size of your database, how many rows in session and session_attribute tables?

[OP]

We have two Trac installations.

trac_dev: session: 534668 rows session_attribute: 1772296 rows

trac_ui: session: 307343 rows session_attribute: 973873 rows

Generally what I notice is a many seconds (8-9 seconds) long delay when visiting the tracker for the first time in a while, and a many seconds (16-20 seconds) long delay when viewing a report for the first time. During the delay when viewing a report, SHOW PROCESSLIST shows:

+-----+----------+-----------+----------+---------+------+-----------+------------------------------------------------------------------------------------------------------+
| Id  | User     | Host      | db       | Command | Time | State     | Info                                                                                                 |
+-----+----------+-----------+----------+---------+------+-----------+------------------------------------------------------------------------------------------------------+
[snip]
| 138 | trac_dev | localhost | trac_dev | Query   |   16 | preparing | DELETE FROM session_attribute WHERE authenticated=0 AND sid IN (                                     |
[snip]
+-----+----------+-----------+----------+---------+------+-----------+------------------------------------------------------------------------------------------------------+

Regards,

[OP]

Oh, oops, I forgot to answer your other question. The cron was introduced to fix the problem with the deadlocks. I would just as soon have Trac deal with it itself if it could.

Also, I imagine these performance problems can pop up in other places, but I haven’t spent a lot of time trying to find and document everywhere that they show up.

[Remy Blank]

Replying to OP:

trac_dev: session: 534668 rows session_attribute: 1772296 rows

trac_ui: session: 307343 rows session_attribute: 973873 rows

That's huge. How many users do you actually have? Can you give us an idea of the number of pages served per hour (or minute)? How many unique clients?

To me, this looks like our purging of old sessions isn't working properly. 500'000 sessions, with a purge time of 90 days, means over 5000 new sessions per day. Sounds like a lot.

[OP]

trac_dev: 11749 users, ~338K PVs[1] over the last 90 days, ~1.52M access log entries[2] over the last 52 days. trac_ui: 6211 users, ~184K PVs over the last 90 days, ~1.47M access log entries over the last 52 days.

[1] The pageview counts are only for clients that get tagged by Google Analytics; there are also many (many many) people that use the RSS feeds and RPC service and I can’t give you very confident numbers for those pageviews.
[2] Access log entries were filtered so that requests to /chrome/ and 404s were discarded, but no other filtering was done. Caveat emptor.

[Christian Boos]

Ok, so this might be worth a further refinement.

I think we should open a follow-up ticket with comment:17, but I don't think it's worth postponing 0.12.2 because of that.