Hadoop and Cloudera: Open Source for the Cloud

With the recent Hadoop World event hosted by Cloudera on October 2, 2009, Cloudera and Hadoop have been getting quite a bit of attention from the media, and the visibility for open source software in the cloud has increased along with them. I didn&apost attend the Hadoop World event, but I heard that it was well attended with solid content. Stephen O&aposGrady from RedMonk did a great summary of the event along with his analysis of the key trends if you want a little more information. The Hadoop World event is just a single point in time; however, the more interesting story in my opinion comes from the Hadoop / Cloudera combination.

Hadoop is an Apache project focused on open source software for reliable, scalable, distributed computing. One of the largest contributors to Hadoop is Yahoo, and Hadoop is part of the solution powering Microsoft&aposs new Bing search engine. According to the Who Uses Hadoop page, other companies using Hadoop include Amazon, Adobe, AOL, Facebook, Google, Hulu, IBM, The New York Times, several universities, and many more.

Cloudera was founded to provide enterprise class support for organizations using the open source Hadoop product, and they recently announced the availability of a new product, Cloudera Desktop, a graphical interface for Hadoop that can be used with internal clusters or clusters running in a public cloud. Cloudera has also put together a strong team of people with deep expertise in open source, data management, search and cloud computing. One of the most recent additions to the Cloudera team is Doug Cutting, a search engine specialist from Yahoo and one of the founders of the Hadoop project. This is a big loss for Yahoo and a huge gain for Cloudera.

Om Malik recently weighed in on Cloudera to compare them to Red Hat. He sees many parallels between what Cloudera is doing now for Hadoop and what Red Hat was doing for Linux in the early days from similarities in the executive team, venture capital funding, technologies and more. Here&aposs how he summarized his comparison:

The big change came this past August, when Doug Cutting left Yahoo and joined Cloudera. Cutting¡¦s involvement is like the icing on the cake, giving the company the ability to corner all the Hadoop talent out there. It also helps that Cloudera has started to make inroads into newer markets, including biotech and retail. ¡§Hadoop is going to find potential markets in any industry where there are large data sets that need complex analysis,¡¨ CEO Olson told me.

I remember talking to Red Hat executives back in the day and listening to their pitch about Linux everywhere, how they were going to go beyond the web community and help drive Linux into other corporate environments and eventually, build a services business around it.

Cloudera is following that same path. It¡¦s developed its own version of Hadoop, one that¡¦s optimized for the needs of large corporations, especially those that prefer a little hand-holding from their suppliers. By giving them this version of Hadoop, Cloudera hopes to make revenue from services. And the timing ¡X the company unveiled Cloudera Desktop at Hadoop World (we are media partners) in New York, an event it organized ¡X is perfect.

Game, set, match for Cloudera. (Quoted from GigaOM)

With cloud computing taking off, the open source Hadoop project is well positioned to become a key part of the many clustered solutions with large, complex data requirements, and Cloudera will be right there for those organizations that need help implementing Hadoop.

URL: http://feedproxy.google.com/~r/IntelBlogs/~3/LN-ZcQmCbXs/

Retorno de Investimento

Na última sexta-feira estive em um evento na UNESP onde foi oficialmente lançado o GRIP Unesp com sete clusters sendo interligados em uma rede de pesquisa. Excelente ver universidades ampliando a capacidade de pesquisa e a dedicação de professores e autoridades para tornar isso realidade. Parabéns em especial ao Professor Sergio Novaes que liderou essa iniciativa, o Núcleo de Computação Científica da Unesp está muito bem projetado e com certeza podemos esperar excelentes resultados.

Nesse evento encontrei alguns conhecidos e uma conversa em especial me chamou a atenção quando discutimos sobre como empresas continuam comprando soluções colocando um peso maior no custo de aquisição e desconsiderando custos de manutenção que embora liquidos e certos não estão explicitos no contrato.

Um exemplo disso é uma solução de automação comercial desenvolvida com o processador ATOM que pelos calculos feitos pela empresa que o comercializa trás uma economia de R$184,00 em valor presente de energia elétrica. Ele me contava que mesmo mostrando os calculos que utilizavam uma vida útil de três anos e as especificações técnicas das soluções concorrentes os clientes davam pouca atenção a esse número. Ocorre que uma solução como essa está na faixa de R$700,00 a R$850,00 e portanto estamos falando de um impacto superior a 20% no melhor cenário.

Você pagaria 10% mais por essa solução? Aparentemente muitas empresas não tem conseguido enxergar isso e o mesmo vale para adoção de notebooks pelas empresas. Uma empresa que use desktops equipados com Pentium Dual Core e monitor CRT pode economizar até 17x (dezesete vezes!) nos gastos com energia se migrar para notebooks. E no caso dos notebooks o aumento de produtividade vem como um bonus nem precisa contabilizar! Veja isso no gráfico ao lado.

Abraços e até a próxima

Comments (0)

URL: http://feedproxy.google.com/~r/IntelBrasil/~3/ELU2RE5CGlQ/retorno_de_investimento.php

Super Meat Boy, a game of Tech, Saws, and Awesome

Hello again,

It&aposs been a while since I last posted (I believe its been about 8 months), but I&aposve been incredibly busy. I&aposve been recently working on a game, due out on WiiWare and PC something in Q1 2010, called Super Meat Boy. For those interested, you can find a trailer here:

Super Meat Boy Trailer

I just recently got back from an event in London showcasing Super Meat Boy at a Nintendo WiiWare event. It was great fun, the game went over well and we are very excited. But blah blah blah lets talk some tech about this meaty game (get it?).

The engine for Super Meat Boy has very much been a labor of love. The game is being developed mainly on two machines, an Intel Core 2 Duo 2.16Ghz housed in my (seemingly soon to be dead) laptop and a Intel Core 2 Quad 2.4Ghz desktop. 100% of the development is being done on these two PCs with the engine constructed in such a way that I link in drivers for different renders, input, core, and audio functionality. So far this method of engine design has worked out perfectly. Each module is abstracted out away from engine and game code and is placed in a seperate library. When I need a Wii Build, I link in my Wii libraries and poof, I have a Wii Build. Same with Xbox, PC, etc, etc, etc. This isn&apost that special really, its actually pretty standard. The specialness of this is due to the threading library I originally wrote for Goo!. Super Meat Boy runs on the second generation Goo! engine that features a self detecting threaded renderer…which is just fancy talk for a threaded renderer that does a little bit of a system test on your machine to determine if A. If it can thread it (i.e. is the processor multicore) and B. If it should thread it (i.e. Is the processor fast enough, is the processing power required by the renderer worth threading the rendering). So far it works very well….it even runs well on my little Acer Aspire One with a little Intel GMA 950 and a 1.6Ghz Atom Processor. It threads the renderer on that little thing and it does surprisingly well…I was impressed. On a machine where it should run a threaded renderer (My Core 2 Quad or Duo or the Xbox 360) it sets up the render threads on its own and goes to town, where on lower end PCs and the Wii, it doesn&apost need to thread the renderer so it doesn&apost. It makes my life easier and it utilizes the processors in a users machine, be it console or PC, to provide a nice, smooth gaming experience. The game runs beautifully at 1920×1440…which again, isn&apost saying too much for a game like Super Meat Boy, as it is a 2D platformer with some pretty lights and replays…but its scalable and that&aposs what makes it special.

Right now, on the Wii in particular, I&aposm working on doing some threaded level loading, because I&aposm not a fan of load times, and neither are gamers. With a game like Super Meat Boy, you die often and you beat levels often, so even the slightest amount of wait is just a pain for gamers. Right now, the load time between levels is I believe is less than half a second, but I don&apost like that it pauses. So, what I&aposm doing is loading level A and when level A is loaded, I&aposll fire up a thread that starts the loading of Level B in the background on a very very low priority thread. Since the average level for a pro (like myself) takes anywhere between 5 and 10 seconds (again, I&aposm very pro, most new players will spend between 10 to 20 seconds on easy levels, and probably about 10 minutes on harder levels…this is how it happened in London anyway:)) to play through, this huge amount of time in between levels will allow plenty of processor time for a low priority thread to load the level in and be able to instantly switch to the new level in between level transitions, while then firing up another thread to load the next level. The system will work just fine, and we&aposll be blessed with the gift of little to no load times…that&aposs something that we haven&apost had since the days of cartridge based games:)

That&aposs it for me now, I must sleep, then get back to work. If you&aposre interested in more game engine threading and design, shoot me an email. If you want to keep up with Super Meat Boy, follow us on twitter: http://twitter.com/SuperMeatBoy.

Good Night!!!!

Tommy

URL: http://feedproxy.google.com/~r/IntelBlogs/~3/yZfDg8OQsHc/