Comments on: Using Facebook’s Thrift with Python and HBase

By: Oscar

Oscar — Tue, 15 Mar 2011 22:17:41 +0000

@Varadharajan I also got that error “TypeError: write() argument 1 must be string or read-only character buffer, not dict” for createTable method

I solved it with

desc = ColumnDescriptor( *{‘name’:'foo:’})
the star translates it into kwargs

By: Varadharajan

Varadharajan — Sat, 26 Feb 2011 10:54:00 +0000

When i configured thrift 0.5.0 and build a client as given here, i got a type error as

“TypeError: write() argument 1 must be string or read-only character buffer, not dict” for createTable method

I solved this by using using client.createTable(“table_name”, [ColumnDescriptor(name="foo:")])

Thought this might be useful for some newbie like me

Cheers,
Varadharajan

By: Sami

Sami — Sun, 13 Feb 2011 10:53:57 +0000

I figured out the anwer to my last post. Under the lib/perl directory after extracting Thrift and running make. You need to run make in the lib/perl directory to install these .pm packages.

Thanks,
Sami

By: Sami

Sami — Sun, 13 Feb 2011 10:26:48 +0000

Hi,

I am trying to use a perl script to connect to my Hbase setup just to perform reads from tables. I have the thrift server running and all. Also compiled thrift successfully and generated the Hbase.pm, Constants.pm and Types.pm files for a perl client. What i cant seem to figure out is where to get the following CPAN packages:

use Thrift::BinaryProtocol;
use Thrift::BufferedTransport;
use Thrift::Socket;

Any help will be appreciated.

Thanks,
Sami

By: adebug

adebug — Fri, 26 Nov 2010 13:54:38 +0000

When I write a client to insert data into table, I found the CPU is just totally 30% on 8 core system.
The bottleneck seems that the client read data from local disk file and insert data into htable are serial, so when read , inserting is suspended, and when insert read is suspended.

After I use multi-thread, the thread is hang on thrift’s hbase.py: recv_mutateRows.

Does thrift doesn’t support multi-thread?

By: Leeward Bound

Leeward Bound — Mon, 15 Nov 2010 04:10:07 +0000

Just wanted to update my previous post –
I’ve noticed that my server’s not bottlenecked on either processor or memory while running the create and delete operations in the above script. I can only assume the socket connections to thrift are the bottleneck; I was primarily interested in the random seek times, so I didn’t bother modifying the script to multithread.

By: Leeward Bound

Leeward Bound — Sun, 14 Nov 2010 18:13:55 +0000

Awesome writeup, thanks so much!

I’ve been playing around with HBase to see if it’s feasible to replace MySQL in our current project; This blog post stopped just a little short of what I needed to know, and I had to figure out how to mutate and delete rows by hand.

Here’s my benchmark code, I hope it might serve someone else!
http://pastebin.com/AuuDCKTi

By: Guru Prasad

Guru Prasad — Tue, 10 Aug 2010 09:50:11 +0000

One suggestion: If your client.py (as described in this blog) is throwing error saying “…… thrift.transport.TTransport.TTransportException: Could not connect to localhost:9090″ . This could be due to using ‘localhost’…Fix: replace localhost with your IP.
Note: you may required to make some changes in you /etc/hosts file

By: Adam

Adam — Fri, 25 Dec 2009 19:28:12 +0000

Hey, Great tutorial – Thanks!.

It didnt work out of the box for me however with latest HBase (0.20.2) and thrift (0.2.0-incubator).
It seems that either the thrift auto-generated code or Hbase thrift API changed somewhat.

in particular, one should use kwargs parameter style when creating ColumnDescriptor:

Instead of this:
desc = ColumnDescriptor( { ‘name’ : ‘foo:’ } )
Use this:
desc = ColumnDescriptor(name=’foo’)

Adam

By: Tim

Tim — Fri, 13 Mar 2009 10:59:23 +0000

@Goran: There’s also Dumbo which lets you do map/reduce on hadoop in python. Though it uses hadoop streaming, not thrift.

http://wiki.github.com/klbostee/dumbo