david wong

Hey! I'm David, a security engineer at the Blockchain team of Facebook, previously a security consultant for the Cryptography Services of NCC Group. I'm also the author of the Real World Cryptography book. This is my blog about cryptography and security and other related topics that I find interesting.

Nodster day 2 posted April 2014

Okay. I'm not gonna talk too much about my new node-webkit project Nodster because I also have a lot of studies to do (exams next week!).

nodster

The problems

  • I'm still not streaming the mp3s properly. I think that I'll never be able to do it through the html5 audio and I need to look up other solutions. The do it yourself approach seems appealing and I'm gonna look into node speaker and node lame.
  • I'm using node google to crawl google and get links. It's not working properly and I'll have to dig into crawling google myself.
  • I'm parsing pages with a regex, it really seems to be slowing everything and I'm gonna look into using a dom parser. I heard about cheerio, is it good?

What does the app do?

  • Right now it's crawling google and avoiding useless websites like youtube, facebook, soundcloud, etc...
  • Then it goes one step deeper and looks for .mp3
  • It checks each mp3 for size (thanks to the headers) and download a bit of each to get metadata.
  • Displays the metadata and a link to play the file
  • If the link is clicked, the mp3 is downloaded to a buffer.mp3 and played.

The main problem is that it's slow, and it's not finding enough links. I could try to parse bing, yahoo and a list of mp3 finders. Or I could maybe try to optimize the requests to google...

Anyway, this app is bringing me a lot of problems to solve and it's pretty interesting :) some people have already forked it and someone has already requested a pull, so if you wanna help. Come and fork it!

Well done! You've reached the end of my post. Now you can leave me a comment or read something else.

Comments

Thibaut

I have some node code to scrape google (that works) with captcha solving support (:p) I may push in on my github if you want to play with it.

As for decoding and playing mp3s, have you gave a look to AudioContext.decodeAudioData ? http://www.html5rocks.com/en/tutorials/webaudio/intro/

For parsing cheerio is not bad as far as I can tell, it works pretty well and perfs have not been a problem for me so far

Thibaut

Ping me on twitter if you want to talk


Leave a comment