Here at Northwest Cadence, we tackle problems large and small. Sometimes all it takes to solve a vexing problem is a new perspective on an existing component. Let me paint you a picture based on some of our experiences…
Our webcast, “A Lap Around Cortana Analytics” clearly made an impact. We were starting to see a marked increase in emails and calls around building intelligent business solutions in Azure. This was encouraging but the Cortana Analytics Suite is so robust and easy to implement that some problems can be solved during the initial conference call with a potential client. I distinctly remember one such call.
Jiang called us with a logging problem. His company was storing log messages in Azure table storage. This worked fine for the last six months but they were running into problems as their data collection efforts expanded in step with their company. Jiang explained, “Our tables cannot handle the log volume anymore. It’s not acceptable.” Jiang was having issues with message throughput. Azure table storage can handle about 2,000 one kilobyte writes per second, per partition but Jiang needed more. I asked, “How long do you need to retain the logs you’re collecting?” Jiang thought about it for a moment and replied, “I’d say within three to five days the logs can be purged.” “Perfect,” I blurted out, “Just use an Event Hub.” Jiang muttered something inaudibly before putting us on hold. When he came back he said, “A queue will not work for us. We need to partition the logs and create separate reports from them.” We assured Jiang that this would not be an issue. Azure Event Hubs are not simple queues. They are massively scalable and can handle more than a gigabyte per second of throughput from millions of data sources, all sending messages concurrently. Event Hubs also have partitions. This allows you to segment data and set read/write policies. By creating security tokens, that can be revoked server side, you can set write access to particular partitions within the Event Hub. By creating consumer groups, you can set read policies for the Event Hub; allowing reports to be generated by consumer group. “Best of all,” I told Jiang, “Event Hubs have a flexible retention policy. When you create it in the Azure portal just use custom create and set the message retention to five or six days.” The phone went silent with a click. We were on hold again. “I think this solution will work well for us. What are our next steps?” We told Jiang that we’d be more than happy to help but since Event Hubs are trivial to setup, he really didn’t need our expertise to do it. With that Jiang thanked us for our help and hung up with a solution in hand.
Event Hub Custom Create
For us that’s what it’s all about. Our business is built on relationships and solutions. This time the power and flexibility of Azure Event Hubs made the answer simple, it just took an outside perspective to see it. Next time, Jiang or someone like him may have a more challenging problem and we hope to be there to see that one through as well.