Dear LazyWeb, I could use some help. I feel like I'm asking the wrong questions, and need some nudges in the right direction.
noodl has been very helpful, I think, but his suggestions appear to be answering the question I'm asking, rather than the one I mean. I think.
We have Mongrel + Rails + Apache + proxy_balancer on Server Q out in the DMZ. Mysql is running on Server Z, inside. There's a firewall rule to allow Q to talk to Z on the mysql port. So far so good.
After N minutes, the firewall times out idle connections between Q and Z. N is configurable, of course, but that doesn't fix anything, because people go home over the weekend, and N will eventually be reached. So increasing N postpones the problem, but doesn't fix it.
The problem is thus. After the N minute timeout is reached, the connections to mysql drop. Subsequent requests to the Rails application do not result in the database connection being reestablished, as expected. (At least, it's what I expected, but perhaps I need to adjust my expectations.) Requests to Rails after this point result in an extended wait period, followed eventually by a proxy timeout. The only way to reestablish the mysql connections (that we've found) is to restart the mongrel cluster.
Functions put in a before_filter to reconnect do not seem to be getting called. Indeed, the before_filter doesn't even seem to be reached. It's as though the hangup is happening in some stage before the before_filter - Rails is trying to contact the database, and is waiting indefinitely for a response.
Placing a reconnect in the before_filter works, and reconnects, as long as the mysql connection is up. (Not useful, but interesting.) However, after N minutes are allowed to expire, and the connections drop, that code does not appear to be getting invoked at all.
So it's entirely possible that I'm asking the wrong questions, but my hope is that one of my knowledgeable readers will see this post and immediately say, Oh, sure, that's the well-known problem that is solved like *this*, and *here* is the question you really should have been asking.
Please? :-)
Just another though, and I hope it's wrong 'cos it's unpleasant.
The docs for the ruby mysql driver here http://www.tmtm.org/en/mysql/ruby/ point out that the reconnect option is off by default. However, ActiveRecord doesn't expose the Mysql object through the API so this uses a ruby 'hack' to reach the private variables of the connection adapter. Just for giggles see if this has any effect (in a before_filter as before).
ActiveRecord::Base.connection.instance_eval {@connection.reconnect = true}
Then bounce the mongrels and wait around. If this fixes the problem (and doesn't cause a mysql connection exhaustion) then the real fix would be to patch AR to call this directly.
noodl
http://dev.rubyonrails.org/ticket/3739
I haven't found any followup but the patch might help.
noodl
I don't know much about Ruby and Rails, et al, so I don't know if there is a fix for Rails to reconnect. I would think it generally should, but I could certainly believe arguments on either side of that.
My knowledge is best down in some lower layers, though. :)
Since you're dealing with a device that's loosing state about the TCP connection, one solution is just to ensure that it will never lose that state. Set the value (in Linux 2.6 kernels, anyway) in /proc/sys/net/ipv4/tcp_keepalive_time to a value in seconds lower than the "N" that is the timeout of your firewall.
This value sets the Linux kernel's timer for sending a keepalive probe on a TCP connection. So, if the system goes for that many seconds without any data being sent in that TCP connection, the kernel will send a packet with zero data with the ACK flag set to essentially ACK the last byte sent (a byte that almost assuredly was already ACKed), (implementations vary a bit on these points...the idea remains that a packet is sent on the TCP connection that doesn't actually transmit any data in the TCP bytestream). This has the effect of making sure that the firewall never times out the TCP session from its state tables since the tcp keepalive timer will be reached and elicit packets on the connection before the firewall times out the session.
Jeff
I'm experiencing the same problem, but we're having a local connection through a unix domain socket. We'll be trying some things out and I'll let you know if we can fix it (or produce something extremely dirty which works :))
It will probably be the latter ;)
Regards,
Jeroen
Maybe add that to your environment.rb file?
ActiveRecord::Base.connection.instance_eval {@connection.reconnect = true}
can not go into environment.rb, because .connection is nil at that point.
I think you can put it at the bottom of your environment.rb and you're ok
I believe this is now an option available in the database.yml:
adapter: mysql
...
reconnect: true
...
http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/MysqlAdapter.html
which also references
http://dev.mysql.com/doc/refman/5.0/en/auto-reconnect.html
DGVZmE diwyiwstvikw, [url=http://snldydmyaehi.com/]snldydmyaehi[/url], [link=http://iheaxepvgygx.com/]iheaxepvgygx[/link], http://blwlfhgqzvsi.com/