Sign up for our newsletter and get the latest big data news and analysis.

Building concurrency in Yahoo! apps with Erlang

From time to time I’ve posted about Erlang on this site. It’s the Good & Plenty of programming languages. Everyone has heard of it, but not many folks like it, and it’s always a little interesting when you see someone at the movie theater order the box they’ve had under the counter since the first run of Episode IV: A New Hope.

Anywho, Dr. Dobb’s ran an article this week on a use of Erlang at Yahoo! (which has experimented with many functional languages for subsystems that support their Internet applications) to rebuild one of their core data-gathering applications

Harvester is a data acquisition component at Yahoo! that acquires data from various providers concurrently through a pull model from drop boxes via various protocols. Feeds can either be scheduled or fetched on demand.

While Harvester was originally written in Perl, Erlang’s high-level concurrency constructs — along with OTP design principles — make it an ideal platform for building reliable, fault tolerant, and scalable applications like Harvester. The resulting service is more scalable, available, reliable, and able to comply with tighter service-level agreements (SLA) on a lighter code base and less expensive development efforts. In this article, we compare how the two systems perform side-by-side on a single box and showcase the higher ROI provided by the Erlang stack.

I’m interested: any Erlang out there in classical HPC applications?


  1. […] from:  Building concurrency in Yahoo! apps with Erlang | This entry is filed under News, Yahoo. You can follow any responses to this entry through the RSS […]

Resource Links: