Recently, I had a real opportunity to work on machines across different data centers (DCs). The task is simple: we’d like to replicate data stored in Kafka, a messaging system developed at LinkedIn, from one DC to another. We measured the transfer throughput and it’s extremely low. Even though there is a 1Gb link between [...]
While working on Kafka, a distributed pub/sub system (more on that later) at LinkedIn, I need to use Zookeeper (ZK) to implement the load-balancing logic. I’d like to share my experience of using Zookeeper. First of all, for those of you who don’t know, Zookeeper is an Apache project that implements a consensus service based [...]
Just came back from the 1st ACM Symposium on Cloud Computing at Indianapolis. The conference is collocated with Sigmod and lasts a day and half. A total of 7 people from LinkedIn were at SOCC and the blog below reflects the notes that we took collectively. There were three keynote speeches, all of which are [...]