Nope, but I will add it to my reading list. Thanks!

We might consider posting it but one of the challenges is what is outlined in that ticket. How do you rendezvous with the Flume agent. We've picked a very specific policy for how to do it which may not be a policy that every organization is willing to adopt.

Yes, you could use a decorator to parse your log file and get back to the structure. I could see where that can become more problematic if you are setting properties on the log record in log4j, but of course it could be done that way. The appender does save me from having to manage log files on the application server, which is a primary motivator for using Flume to begin with. And yes, Flume can be used on as few as 1 machines.:)

We wrote an appender to get better structured content into Flume. If we simply tail the log file then we lose content (unless I'm missing something) like level, parsed timestamp, class name, thread name, etc. As written the appender also picks up location information and properties if present and places that into fields. The purpose being to keep as much logger information structured as possible. The HBASE sink sounds great. I'll have a look!

I believe architects need to contribute some code but it will vary to the amount. It helps to keep them informed about the challenges of crafting the system. Deployment is clearly part of the architecture but I don't separate it from the code. Deployability and manageability are part of delivering a complete solution.