Halo, hari ini saya mau sharing tentang bagaimana cara membuat bot web scrapping dengan NodeJS.

Berawal dari keinginan saya untuk mencoba membuat sebuah aplikasi dimana pada form tertentu, user harus mengambil Kurs USD yang up-to-date dari website BI. Ditambah lagi dengan rasa penasaran saya tentang NodeJS akhirnya saya secara tidak sengaja membuat satu tutorial singkat ini.

The case is.. User ingin mengkonversi dari nilai USD ke IDR namun karena user ini berada di intranet, dia tidak dapat memiliki access ke internet (for secutiry reason). What we need is.. User dapat melihat nilai kurs hari ini dan mengkonversinya ke kurs IDR. Sumber kurs harus dari BI. What we do is.. Server mengakses website BI, mengambil salah satu halaman di sana, mengambil element table kurs, mengkonversi dari table ke array, return value sebagai JSON.

Install the NodeJS:

1. Firstly, we need the NodeJS of course. Download di sini (https://nodejs.org/en/download/) Pilih package yang sesuai (saya menggunakan Windows 64-bit). Install and follow the instruction.
2. Buatlah folder baru untuk pengembangan aplikasinya misal di D:\NodeJS
3. Klik start > All programs > Node.js > Node.js command prompt
4. We need to install a required modules. Modulenya adalah request dan cheerio. Jadi, go to the developing path (D:\NodeJS). Ketik “d: ; cd d:\nodejs”
5. Ketik “npm install request cheerio”
6. Tunggu beberapa saat karena npm akan mengunduh module-module tadi.
7. Setelah selesai, akan muncul directory baru di folder tadi namanya “node_modules”.

Create the code:

1. Create a new file “kursbi.js”
2. Open it and apply a new code:
   
   JavaScript

   var request = require("request"), cheerio = require("cheerio"), url = "http://www.bi.go.id/en/moneter/informasi-kurs/transaksi-bi/Default.aspx", isFloat = ['Value', 'Sell', 'Buy'] //untuk menentukan field mana saja yang harus di parse menjadi Float exclude = ['Graph']; //agar field graph tidak dimasukkan ke JSON request({ url: url, time: true }, function (error, response, body) { if (!error) { var $ = cheerio.load(body), result = {}, lastupdate = $("#ctl00_PlaceHolderMain_biWebKursTransaksiBI_lblUpdate").text(), status = "Success", message = "All data collected", col = []; result = { status: status, message: message, data: { lastupdate: lastupdate, table: [] }, elapsedTime: response.elapsedTime }; $('#ctl00_PlaceHolderMain_biWebKursTransaksiBI_GridView2 tr').each(function(i, tr) { var row = {}; switch(i) { case 0: //untuk memisahkan index 0 (paling atas) adalah header $(tr).find('th').each(function(j, th) { col.push($(th).text().trim()); }); break; default: $(tr).find('td').each(function(j, td) { if(exclude.indexOf(col[j]) !== -1) { //kalau field graph maka di-skip return true; } if(isFloat.indexOf(col[j]) !== -1) { var match = $(td).text().trim().match(/[0-9,.]*/); if(match !== null) { row[col[j]] = parseFloat( match[0].replace(/,/g, '') ); //menghilangkan format digit grouping } else { row[col[j]] = parseFloat( match ); //parsing ke Float } } else { row[col[j]] = $(td).text().trim(); } }); result.data.table.push(row); break; } }); console.log(JSON.stringify(result)); } else { console.log(JSON.stringify({ status: "Failed", message: "We've encountered an error: " + error })); } });

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

   var request = require("request"), cheerio = require("cheerio"), url = "http://www.bi.go.id/en/moneter/informasi-kurs/transaksi-bi/Default.aspx", isFloat = ['Value', 'Sell', 'Buy'] //untuk menentukan field mana saja yang harus di parse menjadi Float exclude = ['Graph']; //agar field graph tidak dimasukkan ke JSON   request({ url: url, time: true }, function (error, response, body) { if (!error) { var $ = cheerio.load(body), result = {}, lastupdate = $("#ctl00_PlaceHolderMain_biWebKursTransaksiBI_lblUpdate").text(), status = "Success", message = "All data collected", col = [];   result = { status: status, message: message, data: { lastupdate: lastupdate, table: [] }, elapsedTime: response.elapsedTime };   $('#ctl00_PlaceHolderMain_biWebKursTransaksiBI_GridView2 tr').each(function(i, tr) { var row = {}; switch(i) { case 0: //untuk memisahkan index 0 (paling atas) adalah header $(tr).find('th').each(function(j, th) { col.push($(th).text().trim()); }); break; default: $(tr).find('td').each(function(j, td) { if(exclude.indexOf(col[j]) !== -1) { //kalau field graph maka di-skip return true; }   if(isFloat.indexOf(col[j]) !== -1) { var match = $(td).text().trim().match(/[0-9,.]*/); if(match !== null) { row[col[j]] = parseFloat( match[0].replace(/,/g, '') ); //menghilangkan format digit grouping } else { row[col[j]] = parseFloat( match ); //parsing ke Float } } else { row[col[j]] = $(td).text().trim(); } }); result.data.table.push(row); break; } });   console.log(JSON.stringify(result)); } else { console.log(JSON.stringify({ status: "Failed", message: "We've encountered an error: " + error })); } });

3. Save it!

Test run:

1. Masih dalam Node.js command prompt tadi ketik “node kursbi.js”
2. Akan terasa NodeJS berhenti sesaat karena NodeJS sedang mengambil page dari web BI lalu mengolahnya.
3. Akan keluar hasil dalam bentuk JSON:
   
   JavaScript

   { "status": "Success", "message": "All data collected", "data": { "lastupdate": "26 February 2016", "table": [{ "Currencies": "AUD", "Value": 1, "Sell": 9751.45, "Buy": 9647.76 }, { "Currencies": "BND", "Value": 1, "Sell": 9619.97, "Buy": 9522.21 }, { "Currencies": "CAD", "Value": 1, "Sell": 9946.09, "Buy": 9844.21 }, { "Currencies": "CHF", "Value": 1, "Sell": 13634.71, "Buy": 13485.38 }, { "Currencies": "CNY", "Value": 1, "Sell": 2061.13, "Buy": 2040.62 }, { "Currencies": "DKK", "Value": 1, "Sell": 1994.9, "Buy": 1974.76 }, { "Currencies": "EUR", "Value": 1, "Sell": 14885.08, "Buy": 14731.63 }, { "Currencies": "GBP", "Value": 1, "Sell": 18838.99, "Buy": 18643.53 }, { "Currencies": "HKD", "Value": 1, "Sell": 1733.65, "Buy": 1716.29 }, { "Currencies": "JPY", "Value": 100, "Sell": 11931.43, "Buy": 11809.57 }, { "Currencies": "KRW", "Value": 1, "Sell": 10.91, "Buy": 10.8 }, { "Currencies": "KWD", "Value": 1, "Sell": 44830.23, "Buy": 44325.13 }, { "Currencies": "MYR", "Value": 1, "Sell": 3204.9, "Buy": 3168.49 }, { "Currencies": "NOK", "Value": 1, "Sell": 1563.84, "Buy": 1547.74 }, { "Currencies": "NZD", "Value": 1, "Sell": 9106.39, "Buy": 9010.44 }, { "Currencies": "PGK", "Value": 1, "Sell": 4417.18, "Buy": 4273.23 }, { "Currencies": "PHP", "Value": 1, "Sell": 283.01, "Buy": 280.13 }, { "Currencies": "SAR", "Value": 1, "Sell": 3592.16, "Buy": 3554.23 }, { "Currencies": "SEK", "Value": 1, "Sell": 1589.01, "Buy": 1572.27 }, { "Currencies": "SGD", "Value": 1, "Sell": 9619.97, "Buy": 9522.21 }, { "Currencies": "THB", "Value": 1, "Sell": 377.54, "Buy": 373.68 }, { "Currencies": "USD", "Value": 1, "Sell": 13467, "Buy": 13333 }] }, "elapsedTime": 1357 }

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140

   { "status": "Success", "message": "All data collected", "data": { "lastupdate": "26 February 2016", "table": [{ "Currencies": "AUD", "Value": 1, "Sell": 9751.45, "Buy": 9647.76 }, { "Currencies": "BND", "Value": 1, "Sell": 9619.97, "Buy": 9522.21 }, { "Currencies": "CAD", "Value": 1, "Sell": 9946.09, "Buy": 9844.21 }, { "Currencies": "CHF", "Value": 1, "Sell": 13634.71, "Buy": 13485.38 }, { "Currencies": "CNY", "Value": 1, "Sell": 2061.13, "Buy": 2040.62 }, { "Currencies": "DKK", "Value": 1, "Sell": 1994.9, "Buy": 1974.76 }, { "Currencies": "EUR", "Value": 1, "Sell": 14885.08, "Buy": 14731.63 }, { "Currencies": "GBP", "Value": 1, "Sell": 18838.99, "Buy": 18643.53 }, { "Currencies": "HKD", "Value": 1, "Sell": 1733.65, "Buy": 1716.29 }, { "Currencies": "JPY", "Value": 100, "Sell": 11931.43, "Buy": 11809.57 }, { "Currencies": "KRW", "Value": 1, "Sell": 10.91, "Buy": 10.8 }, { "Currencies": "KWD", "Value": 1, "Sell": 44830.23, "Buy": 44325.13 }, { "Currencies": "MYR", "Value": 1, "Sell": 3204.9, "Buy": 3168.49 }, { "Currencies": "NOK", "Value": 1, "Sell": 1563.84, "Buy": 1547.74 }, { "Currencies": "NZD", "Value": 1, "Sell": 9106.39, "Buy": 9010.44 }, { "Currencies": "PGK", "Value": 1, "Sell": 4417.18, "Buy": 4273.23 }, { "Currencies": "PHP", "Value": 1, "Sell": 283.01, "Buy": 280.13 }, { "Currencies": "SAR", "Value": 1, "Sell": 3592.16, "Buy": 3554.23 }, { "Currencies": "SEK", "Value": 1, "Sell": 1589.01, "Buy": 1572.27 }, { "Currencies": "SGD", "Value": 1, "Sell": 9619.97, "Buy": 9522.21 }, { "Currencies": "THB", "Value": 1, "Sell": 377.54, "Buy": 373.68 }, { "Currencies": "USD", "Value": 1, "Sell": 13467, "Buy": 13333 }] }, "elapsedTime": 1357 }

Hasil dari JSON ini bisa diambil dan disimpan ke database atau dipreview ke PHP. Tutorial berikutnya saya akan coba membuatnya dengan PHP.

Terima kasih. Semoga bermanfaat.