A port of n0madic/twitter-scraper to Node.js.
Twitter's API is annoying to work with, and has lots of limitations — luckily their frontend (JavaScript) has it's own API, which I reverse-engineered. No API rate limits. No tokens needed. No restrictions. Extremely fast.
You can use this library to get the text of any user's Tweets trivially.
Known limitations:
scraper.login()
.This package requires Node.js v16.0.0 or greater.
NPM:
npm install @the-convocation/twitter-scraper
Yarn:
yarn add @the-convocation/twitter-scraper
TypeScript types have been bundled with the distribution.
Most use cases are exactly the same as in
n0madic/twitter-scraper. Channel
iterators have been translated into
AsyncGenerator
instances, and can be consumed with the corresponding
for await (const x of y) { ... }
syntax.
This package directly invokes the Twitter API, which does not have permissive
CORS headers. With the default settings, requests will fail unless you disable
CORS checks, which is not advised. Instead, applications must provide a CORS
proxy and configure it in the Scraper
options.
Proxies (and other request mutations) can be configured with the request interceptor transform:
const scraper = new Scraper({
transform: {
request(input: RequestInfo | URL, init?: RequestInit) {
// The arguments here are the same as the parameters to fetch(), and
// are kept as-is for flexibility of both the library and applications.
if (input instanceof URL) {
const proxy = "https://corsproxy.io/?" +
encodeURIComponent(input.toString());
return [proxy, init];
} else if (typeof input === "string") {
const proxy = "https://corsproxy.io/?" + encodeURIComponent(input);
return [proxy, init];
} else {
// Omitting handling for example
throw new Error("Unexpected request input type");
}
},
},
});
corsproxy.io is a public CORS proxy that works correctly with this package.
The public CORS proxy corsproxy.org does not work at the time of writing (at least not using their recommended integration on the front page).
"use client";
import { Scraper, Tweet } from "@the-convocation/twitter-scraper";
import { useEffect, useMemo, useState } from "react";
export default function Home() {
const scraper = useMemo(
() =>
new Scraper({
transform: {
request(input: RequestInfo | URL, init?: RequestInit) {
if (input instanceof URL) {
const proxy = "https://corsproxy.io/?" +
encodeURIComponent(input.toString());
return [proxy, init];
} else if (typeof input === "string") {
const proxy = "https://corsproxy.io/?" +
encodeURIComponent(input);
return [proxy, init];
} else {
throw new Error("Unexpected request input type");
}
},
},
}),
[],
);
const [tweet, setTweet] = useState<Tweet | null>(null);
useEffect(() => {
async function getTweet() {
const latestTweet = await scraper.getLatestTweet("twitter");
if (latestTweet) {
setTweet(latestTweet);
}
}
getTweet();
}, [scraper]);
return (
<main className="flex min-h-screen flex-col items-center justify-between p-24">
{tweet?.text}
</main>
);
}
This package currently uses
cross-fetch
as a portable
fetch
. Edge runtimes such as CloudFlare Workers sometimes have fetch
functions that behave differently from the web standard, so you may need to
override the fetch
function the scraper uses. If so, a custom fetch
can be
provided in the options:
const scraper = new Scraper({
fetch: fetch,
});
Note that this does not change the arguments passed to the function, or the
expected return type. If the custom fetch
function produces runtime errors
related to incorrect types, be sure to wrap it in a shim (not currently
supported directly by interceptors):
const scraper = new Scraper({
fetch: (input, init) => {
// Transform input and init into your function's expected types...
return fetch(input, init)
.then((res) => {
// Transform res into a web-compliant response...
return res;
});
},
});
This project currently requires Node 18.x for development and uses Yarn for package management. Corepack is configured for this project, so you don't need to install a particular package manager version manually.
The project supports Node 16.x at runtime, but requires Node 18.x to run its build tools.
Just run corepack enable
to turn on the shims, then run yarn
to install the
dependencies.
yarn build
: Builds the project into the dist
folderyarn test
: Runs the package tests (see Testing first)Run yarn help
for general yarn
usage information.
This package includes unit tests for all major functionality. Given the speed at which Twitter's private API changes, failing tests are to be expected.
yarn test
Before running tests, you should configure environment variables for authentication.
TWITTER_USERNAME= # Account username
TWITTER_PASSWORD= # Account password
TWITTER_EMAIL= # Account email
TWITTER_COOKIES= # JSON-serialized array of cookies of an authenticated session
PROXY_URL= # HTTP(s) proxy for requests (optional)
We use Conventional Commits, and enforce this with precommit checks. Please refer to the Git history for real examples of the commit message format.