segmentation fault - MySQL multithreading app and segfault when reconnect to MySQL into worker thread -

i have multithreading application client mysql , use mysql c-client (libmysqlclient_r). have db connections pool, open connection before create thread workers (pthread_create).

the each worker gets single connection connections pool before starting work , puts pool after finishing work. each worker use it's unique connection.

but, database server overload, , mysql client have errors: mysql " lost connection mysql server during query" or " mysql server has gone away". application make reconnect in worker thread:

my_bool res = mysql_ping(c->mysql); if (res) {     mysql_close(c->mysql);     mysql_thread_end();      c->mysql = mysql_init(null);     mysql_thread_init();                      struct conn_desc *cd = &c->db->cds[c->num];     syslog(log_err, "reconnect :[%s:%d]\t%s\tnew mysql=%x tid=%x\n", cd->host,  cd->port, c->db->default_db_name, c->mysql, pthread_self());      res = mysql_real_connect(c->mysql, cd->host, cd->login, cd->passwd, c->db->default_db_name, cd->port, null, 0);     if (res == null) {         syslog(log_err, "[restart ] reconnect error\n");         exit(1);     } } 

sometime, have segmentation fault mysql_ping() or mysql_real_connect(). why? use separate mysql-connections between workers threads. wrong? how making right?

 0  0x0000000000000000 in ?? ()  1  0x00007ffff7a7fc29 in my_net_local_init () /usr/lib64/mysql/libmysqlclient_r.so.16  2  0x00007ffff7ab0144 in my_net_init () /usr/lib64/mysql/libmysqlclient_r.so.16  3  0x00007ffff7aab245 in **mysql_real_connect ()** /usr/lib64/mysql/libmysqlclient_r.so.16  4  0x000000000040e72c in mysql_query_run (c=0xc36760, q=0x7fffca1fb670 "select     `id`, `name` `msg_dir` `owncrc` = 2831014197") @ mysql.c:163  5  0x000000000040fdf2 in mysql_load_user (uid=2831014197, online=0) @ mysql.c:706  6  0x0000000000406047 in get_mess_count (uid=2831014197, mid=0, online=0) @ commands.c:158  7  0x000000000040618c in cmd_get_all_mess_count (key=0x7fffa80bb074 "gamc|2831014197|0|0 ", data=0x0, data_len=0, ret=0x7fffca1fbbc0, ret_len=0x7fffca1fbbdc) @ commands.c:194  8  0x0000000000405f52 in execute_command (key=0x7fffa80bb074 "gamc|2831014197|0|0 ", data=0x0, data_len=0, ret=0x7fffca1fbbc0, ret_len=0x7fffca1fbbdc) @ commands.c:132  9  0x000000000040c4be in memcache_get (loop=0x7fffa40008c0, mctx=0x7fffa80bb040) @ mc.c:479  10 0x000000000040d353 in memcached_client (loop=0x7fffa40008c0, io=0x7fffa80bb040, revents=1) @ mc.c:785  11 0x00007ffff61e5071 in ev_invoke_pending () /usr/lib64/libev.so.4  12 0x00007ffff61ea23a in ev_run () /usr/lib64/libev.so.4  13 0x000000000040b5ec in ev_loop (loop=0x7fffa40008c0, flags=0) @ /usr/include/libev/ev.h:810  14 0x000000000040e24c in worker_listen (arg=0x10) @ mc.c:1126  15 0x00007ffff762c851 in start_thread () /lib64/libpthread.so.0  16 0x00007ffff5d2f6dd in clone () /lib64/libc.so.6 

and nex bt:

0  0x00000000009f3f70 in ?? () 1  0x00007ffff7aaf32a in net_real_write () /usr/lib64/mysql/libmysqlclient_r.so.16 2  0x00007ffff7aaf63b in net_flush () /usr/lib64/mysql/libmysqlclient_r.so.16 3  0x00007ffff7aaf901 in net_write_command () /usr/lib64/mysql/libmysqlclient_r.so.16 4  0x00007ffff7aac6a9 in cli_advanced_command () /usr/lib64/mysql/libmysqlclient_r.so.16 5  0x00007ffff7a7b1fd in **mysql_ping** () /usr/lib64/mysql/libmysqlclient_r.so.16 6  0x000000000040e8f1 in mysql_query_run (c=0x9ed930, q=0x7fff6fffe670 "select `invisible` meetre.autho2 `crc` = 1032552218") @ mysql.c:164 7  0x00000000004107a0 in mysql_load_user (uid=1032552218, online=1) @ mysql.c:858 8  0x0000000000406278 in get_mess_count (uid=1032552218, mid=0, online=1) @ commands.c:165 9  0x00000000004063bd in cmd_get_all_mess_count (key=0x7fff90383f84 "gamc|1032552218|0|1 ", data=0x0, data_len=0, ret=0x7fff6fffebc0, ret_len=0x7fff6fffebdc) @ commands.c:201 10 0x0000000000406182 in execute_command (key=0x7fff90383f84 "gamc|1032552218|0|1 ", data=0x0, data_len=0, ret=0x7fff6fffebc0, ret_len=0x7fff6fffebdc) @ commands.c:135 11 0x000000000040c718 in memcache_get (loop=0x7fff5c0008c0, mctx=0x7fff90383f50) @ mc.c:459 12 0x000000000040d5cb in memcached_client (loop=0x7fff5c0008c0, io=0x7fff90383f50, revents=1) @ mc.c:765 13 0x00007ffff61e5071 in ev_invoke_pending () /usr/lib64/libev.so.4 14 0x00007ffff61ea23a in ev_run () /usr/lib64/libev.so.4 15 0x000000000040b81c in ev_loop (loop=0x7fff5c0008c0, flags=0) @ /usr/include/libev/ev.h:810 16 0x000000000040e4f4 in worker_listen (arg=0x1e) @ mc.c:1106 17 0x00007ffff762c851 in start_thread () /lib64/libpthread.so.0 18 0x00007ffff5d2f6dd in clone () /lib64/libc.so.6 

code 2:

pthread_mutex_lock(&conn_mutex); my_bool res = mysql_ping(c->mysql); pthread_mutex_unlock(&conn_mutex);  if (res != ok) {     mysql_close(c->mysql);                   mysql_library_end();      pthread_mutex_lock(&conn_mutex);     mysql_library_init(0, null, null);     pthread_mutex_unlock(&conn_mutex);      c->mysql = mysql_init(null);      struct conn_desc *cd = &c->db->cds[c->num];     syslog(log_err, "reconnect :[%s:%d]\t%s\tnew mysql=%x tid=%x %s\n", cd->host,  cd->port, c->db->default_db_name, c->mysql, pthread_self(), mysql_error(c->mysql));     res = mysql_real_connect(c->mysql, cd->host, cd->login, cd->passwd, c->db->default_db_name, cd->port, null, 0);      if (res == null) {         syslog(log_err, "[restart ] reconnect error\n");         exit(1);     } } 

(found via google search. know original author of question long gone, responding posterity in case else stumbles way here.)

your reconnection attempt incredibly overcomplicated, , you're tripping on cause these crashes.

during reconnection attempt, you:

1. release mutex.
2. close connection.
3. allow other threads try use now-closed connection, which segfault. unlocked mutex, after all.
4. call mysql_library_end() without sort of locking, tears down libmysqlclient process-wide. attempt run mysql_ping or mysql_ functions other mysql_connect, no matter or parameters, will segfault, , still don't have lock preventing other threads doing this.
5. you reqacuire lock , call mysql_library_init, , release again reason.
6. finally call mysql_real_connect, telling operate on shared c->mysql variable, without locking. can run multiple threads @ same time, , therefore can segfault.

as can see, there several points code crashworthy. here's how fix it:

1. hold conn_mutex entire operation, before mysql_ping until after reconnect attempt finished. other plan result in threads fighting on gets attempt reconnect.
2. don't call mysql_library_end or mysql_library_init @ -- these calls affecting process-wide state no reason, , have nothing re-establishment of individual connections. call before make first connection in program, , after close last one, , on 1 thread.
3. make sure code calling mysql_thread_init on new threads, , calls mysql_thread_end before exit of threads other main one. (these calls independent of particular connection, , leaving them out can cause subtle issues because things still mostly work. if load data infile segfaults on you, example, problem.)
4. make sure you're linking code thread-safe libmysqlclient_r , not regular libmysqlclient, or none of above save you.

hope helps else similar issues.