Big Data talks in Dhaka 1

Big Data talks in Dhaka
I am spending a few days in Dhaka and am taking the opportunity to meet some companies working on Big Data and give some talks. read more →

GraphChi: How a Mac Mini outperformed a 1,636 node Hadoop cluster

GraphChi: How a Mac Mini outperformed a 1,636 node Hadoop cluster
Last year GraphChi, a spin-off of GraphLab, a distributed graph-based high performance computation framework, did something remarkable. GraphChi outperformed a 1,636 node Hadoop cluster processing a Twitter graph (dataset from 2010) with 1.5 billion edges – using a single Mac Mini. The task was triangle counting and the Hadoop cluster required over 7 hours while ... read more →

ORC: An Intelligent Big Data file format for Hadoop and Hive

ORC: An Intelligent Big Data file format for Hadoop and Hive
RCFile (Record Columnar File), the previous Hadoop Big Data storage format on Hive, is being challenged by the smart ORC (Optimized Row Columnar) format. My first post on the topic, Getting Started with Big Data with Text and Apache Hive, presented a common scenario to illustrate why Hive file formats are significant to its performance and ... read more →

Optimising Hadoop and Big Data with Text and Hive

Optimising Hadoop and Big Data with Text and Hive
Hadoop’s Hive SQL interface reduces costs and to gets results fast with Big Data from Text. Simple optimisations improve the performance significantly. The previous post Getting Started with Big Data with Text and Apache Hive described the case for using text format to import and export data for a Hive ETL and reporting process. These ... read more →

32x Faster Hadoop and Map Reduce With Indexing

32x Faster Hadoop and Map Reduce With Indexing
Hadoop and map reduce’s simplicity, and especially lack of indices, significantly limits its performance. I described how map reduce 2.0 and alternatives bypassing map reduce will change Hadoop’s application and speed it up in the next year or two. Another approach is the introduction of indices to data stored on Hadoop Distributed File System (HDFS). At its inception, ... read more →

Crowd Funding Activism: Put Your Money Where Your ‘Like’ Is

In the Internet age many people utilise technology to spread information, make themselves heard, organise demonstrations, or simply click like to show their support for a cause. The latter is sometimes belittled as the equivalent of a couch potato’s self-gratifying response to a pressing issue. It gives you a warm feeling but has no impact. ... read more →