Node.js, Async/Await and Headless Browsers
If you want to collect data from the web, you’ll come across a lot of resources teaching you how to do this using more established back-end tools like Python or PHP. But there’s a lot less guidance out there for the new kid on the block, Node.js. Thanks to Node.js, JavaScript is a great language to use for a web scraper: not only is Node fast, but you’ll likely end up using a lot of the same methods you’re used to from querying the DOM with front-end JavaScript. Node.js has tools for querying both static and dynamic web pages, and it is well-integrated with lots of useful APIs, node modules and more. In this article, I’ll walk through a powerful way to use JavaScript to build a web scraper. We’ll also explore one of the key concepts useful for writing robust data-fetching code: asynchronous code.Asynchronous Code
Fetching data is often one of the first times beginners encounter asynchronous code. By default, JavaScript is synchronous, meaning that events are executed line-by-line. Whenever a function is called, the program waits until the function is returned before moving on to the next line of code. But fetching data generally involves asynchronous code. Such code is removed from the regular stream of synchronous events, allowing the synchronous code to execute while the asynchronous code waits for something to occur: fetching data from a website, for example. Combining these two types of execution — synchronous and asynchronous — involves some syntax which can be confusing for beginners. We’ll be using theasync and await keywords, introduced in ES7.
They’re syntactic sugar on top of ES6’s Promise syntax, and this — in turn — is syntactic sugar on top of the previous system of callbacks.
Passed-in Callbacks
In the days of callbacks, we were reliant on placing every asynchronous function within another function, leading to what’s sometimes known as the ‘pyramid of doom’ or ‘callback hell’. The example below is on the simple side! /* Passed-in Callbacks */ doSomething(function(result) { doSomethingElse(result, function(newResult) { doThirdThing(newResult, function(finalResult) { console.log(finalResult); }, failureCallback); }, failureCallback); }, failureCallback);Promise, Then and Catch
In ES6, a new syntax was introduced, making it much simpler and easier-to-debug asynchronous code. It is characterised by thePromise object and the then and catch methods:
/* "Vanilla" Promise Syntax */
doSomething()
.then(result => doSomethingElse(result))
.then(newResult => doThirdThing(newResult))
.then(finalResult => {
console.log(finalResult);
})
.catch(failureCallback);
Async and Await
Finally, ES7 broughtasync and await , two keywords which allow asynchronous code to look much closer to synchronous JavaScript, as in the example below.
This most recent development is generally considered the most readable way to do asynchronous tasks in Javascript — and may even boost memory efficiency in comparison to regular Promise syntax.
/* Async/Await Syntax */
(async () => {
try {
const result = await doSomething();
const newResult = await doSomethingElse(result);
const finalResult = await doThirdThing(newResult);
console.log(finalResult);
} catch(err) {
console.log(err);
}
})();
Static Websites
In the past, retrieving data from another domain involved theXMLHttpRequest or XHR object.
Nowadays, we can use JavaScript’s Fetch API.
The fetch() method.
It takes one mandatory argument — the path to the resource you want to fetch (usually a URL) — and returns a Promise .
To use fetch in Node.js, you’ll want to import an implementation of fetch.
Isomorphic Fetch is a popular choice.
Install it by typing npm install isomorphic-fetch es6-promise into the terminal, and then require it at the top of your document like so: const fetch = require('isomorphic-fetch') .
JSON
If you’re fetching JSON data, then you should use thejson() method on your response before processing it:
(async () => {
const response = await fetch('https://wordpress.org/wp-json');
const json = await response.json();
console.log(JSON.stringify(json));
})()
JSON makes it relatively straightforward to grab the data you want from the and process it.
But what if JSON data isn’t available?
HTML
For most websites, you’ll need to extract the data you want from the HTML. With regards to static websites, there are two main ways to go about this. Option A: Regular Expressions If your needs are simple or you’re comfortable writing regex, you can simply use thetext() method, and then extract the data you need using the match method.
For example, here’s is some code to extract the contents of the first h1 tag on a page:
(async () => {
const response = await fetch('https://example.com');
const text = await response.text();
console.log(text.match(/(?<=\<h1>).*(?=\<\/h1>)/));
})()
Option B: A DOM Parser
If you’re dealing with a more complicated document, it can be helpful to make use of JavaScript’s array of in-built methods for querying the DOM: methods like getElementById , querySelector and so on.
If we were writing front-end code, we could use the DOMParser interface.
As we’re using Node.js, we can grab a node module instead.
A popular option is jsdom, which you can install by typing npm i jsdom into the terminal and requiring like this:
const jsdom = require("jsdom");
const { JSDOM } = jsdom;
With jsdom, we can query our imported HTML as its own DOM object using querySelector and related methods:
(async () => {
const response = await fetch('https://example.com');
const text = await response.text();
const dom = await new JSDOM(text);
console.log(dom.window.document.querySelector("h1").textContent);
})()
Dynamic Websites
What if you want to grab data from a dynamic website, where content is generated in real-time, such as on a social media site? Performing afetch request won’t work because it will return the site’s static code, and not the dynamic content that you probably want to get access to.
If that’s what you’re looking for, the best node module for the job is puppeteer — not least because the main alternative, PhantomJS, is no longer being developed.
Puppeteer allows you to run Chrome or Chromium over the DevTools Protocol, with features such as automatic page navigation and screen capture.
By default, it runs as a headless browser, but changing this setting can be helpful for debugging.
Getting Started
To install, navigate to your project directory in the terminal and typenpm i puppeteer .
Here’s some boilerplate code to get you started:
const puppeteer = require('puppeteer');const browser = await puppeteer.launch({
headless: false,
});const page = await browser.newPage();
await page.setRequestInterception(true);await page.goto('http://www.example.com/');
First, we launch puppeteer (disabling headless mode, so we can see what we’re doing).
Then we open a new tab.
The method page.setRequestInterception(true) is optional, allowing us to use abort , continue and respond methods later on.
Lastly, we go to our chosen page.
As in the “DOM Parser” example above, we can now query elements using document.querySelector and the related methods.
Logging In
If we need to log in, we can do so easily using thetype and click methods, which identify DOM elements using the same syntax as querySelector :
await page.type('#username', 'UsernameGoesHere');
await page.type('#password', 'PasswordGoesHere');
await page.click('button');
await page.waitForNavigation();
Handling Infinite Scroll
It is increasingly common for dynamic sites to display content via an infinite scrolling mechanism. To cope with that, you can set puppeteer to scroll down based on certain criteria. Here’s a simple example that will scroll down 5 times, waiting for 1 second between each scroll to account for loading content. for (let j = 0; j < 5; j++) { await page.evaluate('window.scrollTo(0, document.body.scrollHeight)'); await page.waitFor(1000); } Because load times will differ, the above code will not necessarily load the same number of results every time. If that’s a problem, you may want to scroll until a certain number of elements is found, or some other criteria.Making Optimisations
Lastly, there are several ways that you can make optimisations to your code, so that is runs as quickly and smoothly as possible. As an example, here’s a way to get puppeteer to avoid loading fonts or images. await page.setRequestInterception(true);page.on('request', (req) => { if (req.resourceType() == 'font' || req.resourceType() == 'image'){ req.abort(); } else { req.continue(); } }); You could also disable CSS in a similar way, although sometimes the CSS is integral to the dynamic data you want — so I’d err on the side of caution with this one! And that’s pretty much all you need to know to make a functioning web scraper in JavaScript! Once you’ve stored the data in memory, you can then add it to a local document (using thefs module), upload it to a database, or use an API (such as the Google Sheets API ) to send the data directly to a document.
If you’re new to web scraping — or you know about web scraping but you’re new to Node.js — I hope this article has made you aware of some of the powerful tools that make Node.js a very capable scraping tool.
I’d be happy to answer any questions in the comments!
JavaScript Client-side web scraping
scraping-data-with-javascript client-side-web-scraping-with-javascript When I was building my first open-source project, codeBadges, I thought it would be easy to get user profile data from all the main code learning websites. I was familiar with API calls and get requests. I thought I could just use jQuery to fetch the data from the various API's and use it. var name = 'codemzy'; $.get('https://api.github.com/users/' + name, function(response) { var followers = response.followers; }); Well, that was easy. But it turns out that not every website has a public API that you can just grab the data you want from. But just because there is no public API doesn't mean you need to give up! You can use web scraping to grab the data, with only a little extra work. For an example, I will grab my user information from my public freeCodeCamp profile. But you can use these steps on any public HTML page. The first step in scraping the data is to grab the full page html using a jQuery.get request.
var name = "codemzy";
$.get("https://www.freecodecamp.com/" + name, function(response) {
console.log(response);
});
Awesome, the whole page source code just logged to the console.
Note: If you get an error at this stage along the lines of
No ‘Access-Control-Allow-Origin' header is present on the requested resource don't fret.
Scroll down to the Don't Let CORS Stop You section of this post.
That was easy. Using JavaScript and jQuery, the above code requests a page from www.freecodecamp.org, like a browser would. And freeCodeCamp responds with the page. Instead of a browser running the code to display the page, we get the HTML code.
And that's what web scraping is, extracting data from websites.
Ok, the response is not exactly as neat as the data we get back from an API.
But… we have the data, in there somewhere.
Once we have the source code the information we need is in there, we just have to grab the data we need!
We can search through the response to find the elements we need.
Let's say we want to know how many challenges the user has completed, from the user profile response we got back.
At the time of writing, a camper's completed challenges completed are organized in tables on the user profile. So to get the total number of challenges completed, we can count the number of rows.
One way is to wrap the whole response in a jQuery object, so that we can use jQuery methods like .find()
to get the data.
// number of challenges completed
var challenges = $(response).find('tbody tr').length;
This works fine — we get the right result. But its is not a good way to get the result we are after.
Turning the response into a jQuery object actually loads the whole page, including all the external scripts, fonts and stylesheets from that page…Uh oh!
We need a few bits of data. We really don't need the page the load, and certainly not all the external resources that come with it.
We could strip out the script tags and then run the rest of the response through jQuery. To do this, we could use Regex to look for script patterns in the text and remove them.
Or better still, why not use Regex to find what we are looking for in the first place?
// number of challenges completed
var challenges = response.replace(/<thead>[\s|\S]*?<
\/thead>/g).match(/<tr>/g).length;
And it works! By using the Regex code above, we strip out the table head rows (that did not contain any challenges), and then match all table rows to count the number of challenges completed.
It's even easier if the data you want is just there in the response in plain text.
At the time of writing the user points were in the html like
<h1 class=”flat-top text-primary”>[ 1498 ]</h1>
just waiting to be scraped.
var points = response.match(/<h1 class="flat-top text-primary">\[ ([\d]*?) \]<\/h1>/)[1];
In the above Regex pattern we match the h1 element we are looking for including the [ ] that surrounds the points, and group any number inside with ([\d]*?).
We get an array back, the first [0] element is the entire match and the second [1] is our group match (our points).
Regex is useful for matching all sorts of patterns in strings, and it is great for searching through our response to get the data we need.
You can use the same 3 step process to scrape profile data from a variety of websites:
Use client-side JavaScript
Use jQuery to scrape the data
Use Regex to filter the data for the relevant information
Until I hit a problem, CORS.
Don't Let CORS Stop You!
CORS or Cross-Origin Resource Sharing, can be a real problem with client-side web scraping. For security reasons, browsers restrict cross-origin HTTP requests initiated from within scripts. And because we are using client-side Javascript on the front end for web scraping, CORS errors can occur. Staying firmly within our front end script, we can use cross-domain tools such as Any Origin, Whatever Origin, All Origins, crossorigin and probably a lot more. I have found that you often need to test a few of these to find the one that will work on the site you are trying to scrape. Example, we can send our request via a cross-domain tool to bypass the CORS issue. $.getJSON('http://www.whateverorigin.org/get?url=' + encodeURIComponent('http://google.com') + '&callback=?', function(data){ console.log(data.contents); }); Call a external web page (cross-domain) with javascriptWebscraping without Node js
search content of external webpages using javascript (e.g. running a webworker in background). Webscraping without Node js Yes, this is possible. Just use the XMLHttpRequest API: Note that I had to use a proxy to bypass CORS issues; if you want to do this, run your own proxy on your own server. var request = new XMLHttpRequest(); request.open("GET", "https://bypasscors.herokuapp.com/api/?url=" + encodeURIComponent("https://duckduckgo.com/html/?q=stack+overflow"), true); // last parameter must be true request.responseType = "document"; request.onload = function (e) { if (request.readyState === 4) { if (request.status === 200) { var a = request.responseXML.querySelector("div.result:nth-child(1) > div:nth-child(1) > h2:nth-child(1) > a:nth-child(1)"); console.log(a.href); document.body.appendChild(a); } else { console.error(request.status, request.statusText); } } }; request.onerror = function (e) { console.error(request.status, request.statusText); }; request.send(null); // not a POST request, so don't send extra data Puppeteer Puppeteer is a Node library to control Chrome over the DevTools Protocol. Puppeteer runs headless by default, but can be configured to run full (non-headless) Chrome or Chromium.How to scrape the web with JavaScript
Whether you're a student, researcher, journalist, or just plain interested in some data you've found on the internet, it can be really handy to know how to automatically save this data for later analysis, a process commonly known as "scraping". There are several different ways to scrape, each with their own advantages and disadvantages and I'm going to cover three of them in this article: Scraping a JSON API Scraping sever-side rendered HTML Scraping JavaScript rendered HTML For each of these three cases, I'll use real websites as examples (stats.nba.com, espn.com, and basketball-reference.com respectively) to help ground the process. I'll go through the way I investigate what is rendered on the page to figure out what to scrape, how to search through network requests to find relevant API calls, and how to automate the scraping process through scripts written in Node.js. We'll even try outcurl and jq on the command line for a bit.
So without further adieu, let's begin with a quick primer on CSV vs JSON.
Note these instructions were written with Chrome 78 and will likely vary slightly with different browsers.
I know CSV, but what is JSON?
You may be familiar with CSV files as a common way of working with data, but for the web, the standard is to use JSON. There are plenty of tools online that can convert between the two formats so if you need a CSV, getting from one to the other shouldn't be a problem. Here's a brief comparison of how some data may look in CSV vs JSON: ID,Name,Season,Points 1,LeBron James,2019-20,25.2 1,LeBron James,2018-19,27.4 1,LeBron James,2017-18,27.5 CSV [ { "id": 1, "name": "LeBron James", "season": "2019-20", "points": 25.2 }, { "id": 1, "name": "LeBron James", "season": "2018-19", "points": 27.4 }, { "id": 1, "name": "LeBron James", "season": "2017-18", "points": 27.5 } ] JSON So there's a bunch more symbols and the headers are integrated into each item in the JSON version. While this is a more common JSON format, some times we'll see data in other arrangements too, which are more similar to CSV. For example, stats.nba.com uses a format similar to: { "headers": ["id", "name", "season", "points"], "rows": [ [1, "LeBron James", "2019-20", 25.2], [1, "LeBron James", "2018-19", 27.4], [1, "LeBron James", "2017-18", 27.5] ] } JSON Alternative FormCase 1 – Using APIs Directly
A very common flow that web applications use to load their data is to have JavaScript make asynchronous requests (AJAX) to an API server (typically REST or GraphQL) and receive their data back in JSON format, which then gets rendered to the screen. In this case, we'll go over a method of intercepting these API requests and work with their JSON payloads directly via a script written in Node.js. We'll use stats.nba.com as our case study to learn these techniques.Find the relevant API requests
Okay with some preliminary understanding of data formats under our belt, it's time to take a stab at scraping some real data. Our goal will be to write a script that will save LeBron James' year-over-year career stats.Step 1: Check if the data is loaded dynamically
Let's head on over to stats.nba.com and find the page with the stats we care about, in this case LeBron's player page: https://stats.nba.com/player/2544/
Screenshot of LeBron's player page
Hey! There's our table right up top on LeBron's page, how convenient.
We need to check if this data is already in the HTML or if it is loaded dynamically via a JSON API.
To do so, we'll View Source on the page and try to find the data in the HTML.
Right click on the page and select View Source
The source code of LeBron's page
With the source loaded, we can look for the data by searching (via Cmd+F) for some values that show up in it (e.g.
2017-18 or 51.3).
Alas, there are no results! This confirms that the data is loaded dynamically, likely via a JSON API.
Step 2: Find the API endpoint that returns the data
OK, so the data isn't in the HTML, where the heck is it? Let's inspect the element with our browser's developer tools to see if we can find any clues. Right click the table and select Inspect.
Right click on the table and select Inspect
The dynamically generated HTML source for the table
We're in luck! Looks like that tag nba-stat-table has some clues in it.
This doesn't always happen, but it sure helps when it does.
<nba-stat-table
ng-if="!datasets.Base.isLoading && !datasets.Base.noData"
rows="datasets.Base.sets[1].rows"
params="params"
filters="filters"
ai="ai"
template="player/player-traditional">
In particular, it references datasets.Base and datasets.Base.sets[1].rows.
We now have to hunt through the network calls made after the page has loaded to see if we can find the matching API request.
To do so, load up the Network tab in the browser's developer tools and refresh the page to capture all requests.
Note the tools should already be open from when you clicked Inspect to load the HTML inspector.
Network is just another tab in that panel.
Otherwise you can go to View → Developer → Developer Tools to open them.
After refreshing the page, your network tab should be full of stuff as shown below.
The Network tab in Chrome's developer tools
These are all the requests the browser makes to render the page after loading the initial HTML.
Let's try and filter it down to something more manageable to sift through.
Click the filter icon (third from the left for me) to reveal a filter panel.
Select the XHR button to filter out everything (e.g., images, videos) except API requests.
Then let's cross our fingers and try filtering for the word "Base" since that seems to be the dataset we care about based on the HTML we found above.
Filtering the Network tab
Ooo looks like we've got something.
Click the row playerdashboardbyyearoveryear then Response to see what's in it.
The response of playerdashboardyearoveryear
Yikes! That's not very easy to read, but hey it sure looks like JSON.
At this point, I take the text of the response and copy and paste it into a formatter so I can get a better understanding of what's in it.
I use my text editor for this, but googling for an online JSON formatter will work just as well.
{
"resource": "playerdashboardbyyearoveryear",
"parameters": { "MeasureType": "Base", ...
},
"resultSets": [
{ "name": "OverallPlayerDashboard", ...
},
{
"name": "ByYearPlayerDashboard",
"headers": [ "GROUP_SET", "GROUP_VALUE", "TEAM_ID",
"TEAM_ABBREVIATION", "MAX_GAME_DATE", "GP", "W", "L",
"W_PCT", "MIN", "FGM", "FGA", "FG_PCT", ...
],
"rowSet": [
[ "By Year", "2019-20", 1610612747,
"LAL", "2019-11-23T00:00:00", 16, 14, 2,
0.875000, 35.2, 9.700000, 19.800000, 0.489, ...
],
...
]
}
]
}
The JSON response from the API request we found, truncated for readability
Recalling the HTML we inspected from earlier, we were looking for a dataset named "Base" and the second set (sets[1] from before) in it to find our table data.
With some careful inspection, we can see that the second item in the resultSets entry in this response matches the data for our table.
We have now confirmed this is the API request we're interested in scraping.
Download the response data with cURL
Now that we know how to manually find the data we care about, let's work on automating it with a script. We'll be using the terminal (Applications/Utilities/Terminal on a Mac) now to quickly iterate with the tools curl and jq. Note you may need to install jq if you do not already have it. The easiest way is with Homebrew. Runbrew install jq on the command line to get it.
Combining these tools with a bash script is probably sufficient for a bunch of scraping needs, but in this article we'll migrate over to using node.js after figuring out the exact request we want to make.
Ok, so back to the Network tab in the browser's developer tools.
Right click the "playerdashboardyearoveryear" row and select Copy → Copy as cURL.
This will copy something like this to your clipboard:
curl 'https://stats.nba.com/stats/playerdashboardbyyearoveryear?DateFrom=&DateTo=&GameSegment=&LastNGames=0&LeagueID=00&Location=&MeasureType=Base&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PaceAdjust=N&PerMode=PerGame&Period=0&PlayerID=2544&PlusMinus=N&Rank=N&Season=2019-20&SeasonSegment=&SeasonType=Regular+Season&ShotClockRange=&Split=yoy&VsConference=&VsDivision=' \
-H 'Connection: keep-alive' \
-H 'Accept: application/json, text/plain, */*' \
-H 'x-nba-stats-token: true' \
-H 'X-NewRelic-ID: VQECWF5UChAHUlNTBwgBVw==' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36' \
-H 'x-nba-stats-origin: stats' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Sec-Fetch-Mode: cors' \
-H 'Referer: https://stats.nba.com/player/2544/' \
-H 'Accept-Encoding: gzip, deflate, br' \
-H 'Accept-Language: en-US,en;q=0.9,id;q=0.8' \
-H 'Cookie: a_bunch_of_stuff=that_i_removed;' \
--compressed
If you paste that into your terminal, you'll see the JSON response show up.
Nice!! But boy there's a bunch of stuff that we're adding to this command: HTTP Headers with the -H flag.
Do we really need all of that? It turns out the answer is maybe.
Some web servers will take precautions to protect against bots, while still allowing normal users to access their sites.
To appear as a normal user while we scrape, we sometimes need to include all kinds of HTTP headers and cookies.
I'd like the request to be as simple as possible, so at this point I try and remove as many of the extra options while still having the request work.
This results in the following:
curl 'https://stats.nba.com/stats/playerdashboardbyyearoveryear?DateFrom=&DateTo=&GameSegment=&LastNGames=0&LeagueID=00&Location=&MeasureType=Base&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PaceAdjust=N&PerMode=PerGame&Period=0&PlayerID=2544&PlusMinus=N&Rank=N&Season=2019-20&SeasonSegment=&SeasonType=Regular+Season&ShotClockRange=&Split=yoy&VsConference=&VsDivision=' \
-H 'Connection: keep-alive' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36' \
-H 'x-nba-stats-origin: stats' \
-H 'Referer: https://stats.nba.com/player/2544/' \
--compressed
Now we can get some nice colors and formatting by simply piping the result to jq: curl ...
| jq.
But jq is awesome and powerful and let's us do a lot more.
For example, we can ask jq to get us the top row in the second result set by running the curl command piped with curl ...
| jq .resultSets[1].rowSet[0]:
curl 'https://stats.nba.com/stats/playerdashboardbyyearoveryear?DateFrom=&DateTo=&GameSegment=&LastNGames=0&LeagueID=00&Location=&MeasureType=Base&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PaceAdjust=N&PerMode=PerGame&Period=0&PlayerID=2544&PlusMinus=N&Rank=N&Season=2019-20&SeasonSegment=&SeasonType=Regular+Season&ShotClockRange=&Split=yoy&VsConference=&VsDivision=' \
-H 'Connection: keep-alive' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36' \
-H 'x-nba-stats-origin: stats' \
-H 'Referer: https://stats.nba.com/player/2544/' \
--compressed | jq .resultSets[1].rowSet[0]
[ "By Year", "2019-20", 1610612747, "LAL", "2019-11-23T00:00:00", 16,
14, 2, 0.875, 35.2, 9.7, 19.8, 0.489, 1.9, 5.6, 0.333, 3.9, 5.6,
0.708, 1, 6.6, 7.6, 10.8, 3.5, 1.3, 0.6, 0.8, 1.4, 4.5, 25.2, 9.7,
52.7, 13, 5, 17, 17, 17, 1, 17, 14, 8, 11, 2, 2, 11, 17, 17, 14, 14,
5, 7, 1, 7, 15, 16, 9, 17, 15, 16, 2, 6, 15, 6, 264, "2019-20" ]
The output of curl ...
| jq .resultSets[1].rowSet[0]
Look at us go! Everything is all right, everything is automatic.
But we miss JavaScript, so let's head on over to Node land.
Write a Node.js script to scrape multiple pages
At this point we've figured out the URL and necessary headers to request the data we want. Now we have everything we need to write a script to scrape the API automatically. You could use whatever language you want here, but I'll do it using node.js with the request library. In an empty directory, run the following commands in your terminal to initialize a javascript project: npm init -y npm install --save request request-promise-native Now we just need to translate our request from the cURL format above to the format needed by the request library. We'll use the promise version so we can take advantage of the very convenient async/await syntax, which let's us work with asynchronous code in a very readable way. Put the following code in a file called index.js. const rp = require("request-promise-native"); const fs = require("fs"); async function main() { console.log("Making API Request..."); // request the data from the JSON API const results = await rp({ uri: "https://stats.nba.com/stats/playerdashboardbyyearoveryear?DateFrom=&DateTo=&GameSegment=&LastNGames=0&LeagueID=00&Location=&MeasureType=Base&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PaceAdjust=N&PerMode=PerGame&Period=0&PlayerID=2544&PlusMinus=N&Rank=N&Season=2019-20&SeasonSegment=&SeasonType=Regular+Season&ShotClockRange=&Split=yoy&VsConference=&VsDivision=", headers: { "Connection": "keep-alive", "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36", "x-nba-stats-origin": "stats", "Referer": "https://stats.nba.com/player/2544/" }, json: true }); console.log("Got results =", results); // save the JSON to disk await fs.promises.writeFile("output.json", JSON.stringify(results, null, 2)); console.log("Done!") } // start the main script main(); Now run this on the command line with node index.js Boom, we replaced a 1-liner from the command line with 30 lines of JavaScript. We're so modern!Getting this data for other players
Cool, so we've got LeBron's data being downloaded from the API automatically, but how about we get it going for a few other players too, so we can feel real productive. You'll notice that the API URL includes a PlayerID query parameter: https://stats.nba.com/stats/playerdashboardbyyearoveryear?DateFrom=&DateTo=&GameSegment=&LastNGames=0&LeagueID=00&
Location=&MeasureType=Base&Month=0&OpponentTeamID=0&Outcome=&
PORound=0&PaceAdjust=N&PerMode=PerGame&Period=0&
PlayerID=2544&
PlusMinus=N&Rank=N&Season=2019-20&SeasonSegment=&
SeasonType=Regular+Season&ShotClockRange=&Split=yoy&
VsConference=&VsDivision= We just need figure out the IDs of the other players we're interested in, modify that parameter and we'll be good to go. I opened a few player pages on stats.nba.com and took the number from the URL to map these IDs to players:
Screenshot of a box score on ESPN
To do so, we'll view the source of the page we're investigating, as we did in Case 1, and search for some of the data (Cmd+F).
In this case, I searched for the remarkable 0-11 stat from Joel Embiid's row.
Screenshot of the server-rendered HTML for the box score
There it is! Right in the HTML, practically begging to be scraped.
Right click the table and select Inspect
You should end up seeing something similar to what is shown below:
The HTML for the table
What we want is a selector that will give us all rows (or 
Steph Curry's 2019-20 Shot Chart from Basketball Reference
By right clicking and inspecting one of the ● or × symbols (similar to how we did it in cases 1 and 2), we can see they are nested