Fark TotalFark  

Fark.com status page

This is the status page for Fark's behind-the-scenes geekery. If there's a major outage or problem, we'll try to post something here about it (unless we're too up to our eyeballs in trying to actually fix it)... so check here if Fark's hosed for an extended length of time. Normally this page will be blank. :-) For more minor issues, see Mike's blog, or more likely, Mike's twitter, Drew's twitter, Fark twitter or all Fark staff/mod/admin twitters together. On (very!) rare occasion you may be able to catch Mike on the data center webcam.

Our normal hosting is at QX.net. This is hosted at a different ISP in case of backbone-related problems -- thanks to Rackspace Cloud for that.


Mon Dec 15 18:47:55 EST 2014: We're looking into some sporadic performance problems, mostly affecting mobile apps. It's proving to be an extremely difficult problem to troubleshoot, and I don't yet have an ETA, however at least one aspect of it should be a bit better as of Mon Dec 15 21:32:49 EST 2014, and fixed for realsies around Mon Dec 15 22:34:44 EST 2014.
Postmortem: The issue was that our webservers, somewhat at random, would each try to allocate about 70 GB of virtual memory, which doesn't go particularly well on machines with 24 GB of physical memory. The randomness of it, plus the fact that the process would just hang make the problem quite difficult to diagnose (ktrace doesn't show syscalls if the process isn't doing anything...) There is a safety net that's supposed to automatically kill any process that just starts malloc'ing memory like crazy, and apparently that wasn't working either. So, I had to fix that first, so that I could get some core dumps to actually trace the wild memory allocation problems. They ended up being tracked down to a 3rd party software module, which had just had a minor update that introduced some serious bugs. It has now been downgraded again. And with the safety nets now working again, this should be less likely to recur in the future should some 3rd party module lose its marbles again.


If you are seeing a delay before Fark starts to load (or Google, Facebook, or other sites since June 6, 2012), see http://test-ipv6.com/ or http://ipv6eyechart.ripe.net/ to see if you have IPv6 connectivity issues that might be causing the issues. If you are IPv4 only, then this test should say "When a publisher offers both IPv4 and IPv6, your browser appears to be happy to take the IPv4 site without delay." If it doesn't, you might... well... have delays. :) You might also try http://netalyzr.icsi.berkeley.edu/ which can diagnose many problems with both IPv4 and IPv6 connections.


Mike the Server Monkey having a bad day

Copyright © 1999-2014 Fark, Inc