Hey there. The pattern in TFA is somewhat different: in the SWR pattern not every worker talks to the DB. Instead, only the selector does. It then hands out the work to the workers via a fast local queue. The ratio I set up for Ping Brigade is 1:1000 selector to workers. Thus a handful of selectors can feed a few thousand workers.
This may be working great in your application but you're implying that it scales nicely and you're wrong about that - hand waving may work in your case, but Baron's whole point was that there are easy solutions that will make things better if and when an application grows to the point at which it's an issue.
I've personally been down this road many times, and the last time I made the mistake of relying on SELECT FOR UPDATE in a queueing system it broke down somewhere on the road between 1msgs/sec and 50msgs/sec. That application committed before it dispatched to the worker app so I would consider it a fairly similar access pattern as yours.
The solution I went with in that case was exactly what Baron describes at "Locking is actually quite easy to avoid." - something along the lines of UPDATE queue SET selected_by = dispatcher_id, selected_time = NOW().. and then SELECT * FROM queue WHERE selected_by = dispatcher_id. I hate putting pseudo-SQL because it's already setting bad ideas in some random reader's head. Anyways, that scaled up to several thousand messages per second and ran happily for years, long after I left that particular company. May still be running depending on who you ask.
Long story short, it's great that your solution is working for you but the weight of public knowledge suggests it's not a great solution for anyone else to pick up on. Ping Brigade looks nifty, I hope it works great for you. Please don't suggest this pattern to other people.
Personally the system I work on day-to-day these days runs a Redis set-based queue similar to Resque to send a few thousand emails per second and I'm ok with it. Not thrilled, but happy enough that I don't read the Resque introduction text and blanch in horror as I did reading your article, especially as a reply to Baron's which is based on... lots and lots of real world experience with many different applications.
Fair points. This solution works for me for now and I certainly know it is not limitless. The solution with setting selected_by is something I thought about and may implement at a later point. I also am a big fan of using GET_LOCK(), for locking rather than relying on MySQL/InnoDB's built-in locking since you have finer grained control over timeouts, etc.
I understand your concern about sharing this "dangerous" knowledge, but I disagree that the solution is to hide it in a deep dark place. Would you find it acceptable if I updated my post with a discussion on scalability and a link to TFA? That way a reader will get more information about building such systems, not less.
The point of the UPDATE .. SELECT if updated pattern is that it's a mark-and-sweep completely outside of a transaction. Avoid locking > lock as little as possible > lock as quickly as possible.
Your blog is your business, the Lorax speaks only for HN.. (hey is Internet Lorax a job?) It would of course be great to update your readers, it'd be cooler to update your application and tell everyone how it worked out! Then you've got a story you can actually submit again
