[Thrift] Compact BinaryProtocol

[Eric Baldeschwieler]

Hi Folks,

A) Great to see the apache proposal.  This is exciting!

B) On the uses of RecordI/O in Hadoop / ZooKeeper / Yahoo:

I'd like to port ZooKeeper to Thrift.  Unfortunately one of our  
biggest users is running on BSD4 in C.  Is anyone working on a Thrift  
C client?  (No Boost, ...)  Anyone interested in working directly on  
Porting ZooKeeper to Thrift with C support?  That would be very  
interesting.

RecordI/O is not currently a serious contender for Hadoop RPC IMO.   
Yahoo is not putting any energy into that, nor are any other  
contributors.  How is Thrift working as a server layer in Java for  
bigTable?  Is anyone else running BIG, highly multi-threaded, server  
code in Java on thrift?  Think 30k clients.  I'd love to hear about  
folks' experience in that area.  When a compelling case can be made  
that moving to Thrift will improve Hadoop performance and stability,  
that would be compelling.  None of the meta-data issues are  
requirements, IMO.  We'd be happy to do some performance and scale  
testing of a serious patch!

In terms of meta-data management, our big requirement is simply to  
have a clear human and machine readable meta-data record that sits  
besides a data collection.  We want to be able to put these record in  
DBs / meta-data repositories and then query across large collections  
of records that have similar but not identical schemas (think all  
recorded logs in a system that has seen 12 revisions).  Something  
like Pig wants to consider all the metadata and make a query plan  
before scanning the records.

---

Hope this helps!

---
eric14 a.k.a. Eric Baldeschwieler
grid computing
Yahoo!  Inc.