david wong

Hey! I'm David, a security consultant at Cryptography Services, the crypto team of NCC Group . This is my blog about cryptography and security and other related topics that I find interesting.

Nodster day 2

posted April 2014

Okay. I'm not gonna talk too much about my new node-webkit project Nodster because I also have a lot of studies to do (exams next week!).

nodster

The problems

  • I'm still not streaming the mp3s properly. I think that I'll never be able to do it through the html5 audio and I need to look up other solutions. The do it yourself approach seems appealing and I'm gonna look into node speaker and node lame.
  • I'm using node google to crawl google and get links. It's not working properly and I'll have to dig into crawling google myself.
  • I'm parsing pages with a regex, it really seems to be slowing everything and I'm gonna look into using a dom parser. I heard about cheerio, is it good?

What does the app do?

  • Right now it's crawling google and avoiding useless websites like youtube, facebook, soundcloud, etc...
  • Then it goes one step deeper and looks for .mp3
  • It checks each mp3 for size (thanks to the headers) and download a bit of each to get metadata.
  • Displays the metadata and a link to play the file
  • If the link is clicked, the mp3 is downloaded to a buffer.mp3 and played.

The main problem is that it's slow, and it's not finding enough links. I could try to parse bing, yahoo and a list of mp3 finders. Or I could maybe try to optimize the requests to google...

Anyway, this app is bringing me a lot of problems to solve and it's pretty interesting :) some people have already forked it and someone has already requested a pull, so if you wanna help. Come and fork it!

Well done! You've reached the end of my post. Now you can leave me a comment :)

Thibaut

I have some node code to scrape google (that works) with captcha solving support (:p) I may push in on my github if you want to play with it.

As for decoding and playing mp3s, have you gave a look to AudioContext.decodeAudioData ? http://www.html5rocks.com/en/tutorials/webaudio/intro/

For parsing cheerio is not bad as far as I can tell, it works pretty well and perfs have not been a problem for me so far

Thibaut

Ping me on twitter if you want to talk