A tutorial on using PeerJs in node-webkit app

In this article, I'm going to talk about the combination usage of node-webkit and PeerJs, but first let's take a look at things we're going to talk about. Node-webkit is an app runtime that allows developers to use Web technologies (i.e. HTML5, CSS and JavaScript) to develop native apps. PeerJS wraps the browser's WebRTC implementation to provide a complete, configurable, and easy-to-use peer-to-peer connection API. If you haven't heard about them, I suggest you go to their websites and take a quick look at what they do, cause they both are really insteresting projects.

Why do I write this article?

There has been some discussions on running peerjs client in a node.js application, but apparently it's not easy to do this because PeerJs relies on WebRTC which is built into Webkit. Then, as node-webkit gets more and more popular, people start to think, why not use PeerJs in node-webkit so that we can build p2p apps? Here is an attempt, as you see it really works, basically it's the same as running PeerJs in browser.

Yet, simply using node-webkit as a browser and running PeerJs in it waste the most powerful feature node-webkit provides: the node.js runtime. Browser is cool, HTML5 give us the ability to cope with files stored in local computer, but those features are minor compared to what nodejs can do. If a node-webkit app don't make use of nodejs, why bother using node-webkit instead of writing a pure web app?

The node-webkit project I'm working on needs nodejs(db strorage, watching files, etc...) as well as PeerJs. First I tried something like this:

<!DOCTYPE html>
<html>
<head lang="en">
  <script src="peer.js"></script>
</head>
<body>
<script>
  var peer = new Peer('username', {key: 'my-key'});
  var conn = peer.connect('hehe');
  var fs = require('fs');

  // sender side code
  conn.on('open', function(){
    console.log("connect to peer");
    var data = fs.readFileSync('file-you-want-to-transfer');
    conn.send(data);
    console.log('data sent');
  });

  // receiver side code
  peer.on('connection', function(conn) {
    conn.on('data', function(data){
      fs.writeFileSync('received_file', data);
      console.log("received complete: ", Date());
    });
  });
</script>
</body>
</html>

It doesn't work because there's no node.js runtime in html, So you can't read from/write to local using fs.write/readFileSync. You probabaly know that node-webkit's node runtime can't really interact with DOM environmen——it lets you require a nodejs's script and call its function from DOM, but you CAN'T GET THE RETURN VALUE, the functon you invoked is run in node runtime and knows nothing about DOM, that's why the above code couldn't work.

Then my friend suggested me to run an express server on localhost and use socket.io to make node and DOM interact with each other. I've written a demo and put it on github to show how this works:

https://github.com/laike9m/peerjs-with-nodewebkit-tutorial

This demo could be run either on a single machine or two machines. What it does is transfering .gitignore file to the other end. Here's its GUI:

To run this app, npm install first, then launch it following node-webkit's documentation. Assume you only have one computer, click the receive button, then click send button, you'll see a new file called received_gitignore appear in app's directory. Be sure to click receive before send, whether running on single machine or two machines.

Finally, let's get down to business to explain how this demo works.

First is package.json:

{
    "main": "main.html",
}

So main.html is the first HTML page node-webkit should display.

<html>
<script>
  var main = require('./main.js');
  window.location.href = "http://127.0.0.1:12345/";
</script>
</html>

Here's the interesting part: our main.html doesn't contain anything to display, it's only purpose is calling require('./main') which will launch an express server listening on 127.0.0.0:12345, then connects to it.

Let's see how it's done in main.js:

// main.js part 1
var app = require('express')();
var server = require('http').Server(app);
var io = require('socket.io')(server);

server.listen(12345);
app.use(require('express').static(__dirname + '/static'));
app.get('/', function (req, res) {
  res.sendfile(__dirname + '/index.html');
});

Nothing special, just a regular express http server with socket.io.
As we can see, visiting http://127.0.0.1:12345/ gets index.html displayed. Here's index.html:

<!DOCTYPE html>
<html>
<head lang="en">
  <meta charset="UTF-8">
  <meta name="author" content="laike9m">
  <title>demo</title>
  <script type="text/javascript" src="peer.js"></script>
  <script type="text/javascript" src="/socket.io/socket.io.js"></script>
</head>
<body>
  <script>
    window.socket = io.connect('http://localhost/', { port: 12345 });
    function clickSend() {
      var peer = new Peer('sender', {key: '45rvl4l8vjn3766r'});
      var conn = peer.connect('receiver');
      conn.on('open', function () {
        console.log("sender dataconn open");
        window.socket.on('sendToPeer', function(data) {
          console.log("sent data: ", Date());
          conn.send(data);
          peer.disconnect();
        });
        window.socket.emit('send');
      });
    }
    function clickRecv(){
      var peer = new Peer('receiver', {key: '45rvl4l8vjn3766r'});
      peer.on('connection', function(conn) {
        conn.on("open", function(){
          console.log("receiver dataconn open");
          conn.on('data', function(data){
            console.log("received data: ", Date());
            window.socket.emit('receive', data);
          });
        });
      });
    }
  </script>
  peerjs with nodewebkit demo
  <button onclick="clickSend()">send</button>
  <button onclick="clickRecv()">receive</button>
</body>
</html>

It contains two button: send and receive. When clicked, clickSend and clickRecv gets called. To understand what these functions do, let's see the other part of main.js.

// main.js part 2
io.on('connection', function(socket){
  socket.on('send', function(data){
    socket.emit('sendToPeer', fs.readFileSync('.gitignore'));
  });
  socket.on('receive', function(data){
    fs.writeFileSync('received_gitignore', data);
  });
});

So you clicked the receive button, PeerJs create a Peer with id receiver and a valid key I registered, then it waits for connection from sender. Then you clicked the send button, another Peer is created with id sender. sender tries to connect to receiver and when data connection successfully built, window.socketemits a send event, which is handled in main.js. The handler simply reads file content and sends it back to DOM environment. Note that socket.io supports binary data transfer from version 1.0, so you can't use a lower version socket.io.

Coming back to code in clickSend, we've already created an event handler on sendToPeer before emitting send event, now it gets fired. conn.send(data) sends data to Peer receiver. In function clickRecv, when conn receives data, it uses window.socket to emit a receive event and sends data from sender to Node runtime. Finally fs.writeFileSync('received_gitignore', data) write data to disk, all done.

You might wonder if it actually works for large file transfer. It does. My project is working great, it can transfer large files with decent speed, the prototype is this demo. Of course you should write many many more lines to make this prototype a usable app, for instance, data needs to be sliced when transfering large files, and you should handle all kinds of PeerJs errors.

That's all for this tutorial.

互联网公司GitHub repo 语言使用情况

做 PPT 太无聊了,突然想到可以统计一下这个东西,于是就做了一下
现在基本上所有国外大公司和国内部分公司都在 GitHub 上开源了一部分代码。统计一下这些代码的语言使用情况,多少可以反映公司内部对语言的偏好。很多公司流行的项目都是单独建一个 repo的,没办法统计,所以这里统计大家就随便看看吧。使用了 GitHub 的 API,只有不到四十行代码,所以直接贴在这里了,复制下来装个 requests 就可以直接运行

# coding: utf-8

"""
统计大公司github上的organization 中repo 的语言使用情况
"""

import requests
from collections import defaultdict
from os.path import join
from pprint import pprint


class GetLangStat():

    api_url = "https://api.github.com/orgs"
    ORGANIZATIONS = (
        'Microsoft', 'aws', 'google', 'twitter', 'facebook',
        'alibaba'
    )
    stats = {org: defaultdict(int) for org in ORGANIZATIONS}

    @classmethod
    def get_one_org_repos(cls, org):
        print(org)
        url = join(cls.api_url, org, 'repos')
        r = requests.get(url)
        for repo in r.json():
            cls.stats[org][repo['language']] += 1

    @classmethod
    def get_all_org_repos(cls):
        for org in cls.ORGANIZATIONS:
            cls.get_one_org_repos(org)
        pprint(cls.stats)


if __name__ == '__main__':
    GetLangStat.get_all_org_repos()

统计的公司包括 MS,amazon,google, twitter, facebook, 阿里。其中 amazon 似乎只开源了 aws 相关的代码,不过也算进来了。本来想找百度和腾讯的,结果发现百度没有一个统一的 organization,都是按产品散着的,腾讯则基本没有开源代码。。。

下面是统计结果,每个公司只取前五名

阿里

Language repo count
Java 13
C 8
C++ 2
JavaScript 2
Perl 2

google

Language repo count
JavaScript 9
C++ 4
Ruby 4
Python 3
Java 3

twitter

Language repo count
Scala 14
Ruby 9
Java 3
Python,CSS,JavaScrit,Shell 1

facebook

Language repo count
Java 8
C++ 5
PHP 4
C 3
Python, Js, Objective-C 2

aws

Language repo count
Java 8
Ruby 6
PHP 4
JavaScript 3
Objective-C 3

咳咳,最后是大微软,说实话我也不确定要不要把微软算成互联网公司。。。

Language repo count
C# 29
C++ 1

从这个非常不靠谱的统计来看,Java 果然还是最流行的语言啊。。。不会 Java 感觉压力真的好大 orz

昨天美剧下架,今天射手人人影视关闭

列举一下近几年文化方面的事件吧

英语降分
美剧下架(有版权)
在线漫画网站持续被关(2013.6那次我印象最深因为给其中一个网站写过漫画下载客户端,前不久老店178漫画也没了)
天闻角川出版停滞
肘子被封杀,带鱼成为“民族英雄”
射手网、人人影视关闭

从英语降分开始,我大概就知道现在的政策是怎么回事了。看着吧,除了一些没发法受众太广经济影响太大的地方,比如好莱坞电影,文化方面的限制会继续收紧。想了想还没被动过的领域,音乐是一个,动画是一个。当然音乐不好管,盗版漫天飞,但是动画比较好管。我预测两年内b站乐视爱奇艺不会再买新番版权,b站不允许播放新番,极影关闭。希望到时候事实能打我的脸。

不过话说回来国外看这些东西也很难,只是原因不一样罢了


top