soxoj/tiktok-scraperСканируйте и загружайте посты видео TikTok, профили пользователей, тренды хэштегов и метаданные музыкальной ленты через CLI или модуль Node.js. Поддерживает пакетную загрузку, ротацию прокси и управление сессиями.
Сбор и загрузка полезной информации из TikTok.
Это не официальная поддержка API и т.д. Это просто парсер, использующий веб-API TikTok для сбора медиа и связанной метаинформации.
Подробные описания каждого домена можно найти в соответствующем разделе.```sh
yarn build
tiktok-scraper требует Node.js v10+ для работы.
Установка из NPM```sh npm i -g tiktok-scraper
**Установка из YARN**```sh
yarn global add tiktok-scraper
$ tiktok-scraper --help
Usage: tiktok-scraper [options]
Commands: tiktok-scraper user [id] Scrape videos from username. Enter only username tiktok-scraper hashtag [id] Scrape videos from hashtag. Enter hashtag without # tiktok-scraper trend Scrape posts from current trends tiktok-scraper music [id] Scrape posts from a music id number tiktok-scraper video [id] Download single video without the watermark tiktok-scraper history View previous download history tiktok-scraper from-file [file] [async] Scrape users, hashtags, music, videos mentioned in a file. 1 value per 1 line
Options: --version Show version number [boolean] --session Set session cookie value. Sometimes session can be helpful when scraping data from any method [default: ""] --session-file Set path to the file with list of active sessions. One session per line! [default: ""] --timeout Set timeout between requests. Timeout is in Milliseconds: 1000 mls = 1 s [default: 0] --number, -n Number of posts to scrape. If you will set 0 then all posts will be scraped [default: 0] --since Scrape no posts published before this date (timestamp). If set to 0 the filter is deactived [default: 0] --proxy, -p Set single proxy [default: ""] --proxy-file Use proxies from a file. Scraper will use random proxies from the file per each request. 1 line 1 proxy. [default: ""] --download, -d Download video posts to the folder with the name input [id] [boolean] [default: false] --asyncDownload, -a Number of concurrent downloads [default: 5] --hd Download video in HD. Video size will be x5-x10 times larger and this will affect scraper execution speed. This option only works in combination with -w flag [boolean] [default: false] --zip, -z ZIP all downloaded video posts [boolean] [default: false] --filepath File path to save all output files. [default: "/Users/karl.wint/Documents/projects/javascript/tiktok-scraper"] --filetype, -t Type of the output file where post information will be saved. 'all' - save information about all posts to the` 'json' and 'csv' [choices: "csv", "json", "all", ""] [default: ""] --filename, -f Set custom filename for the output files [default: ""] --noWaterMark, -w Download video without the watermark. NOTE: With the recent update you only need to use this option if you are scraping Hashtag Feed. User/Trend/Music feeds will have this url by default [boolean] [default: false] --store, -s Scraper will save the progress in the OS TMP or Custom folder and in the future usage will only download new videos avoiding duplicates [boolean] [default: false] --historypath Set custom path where history file/files will be stored [default: "/var/folders/d5/fyh1_f2926q7c65g7skc0qh80000gn/T"] --remove, -r Delete the history record by entering "TYPE:INPUT" or "all" to clean all the history. For example: user:bob [default: ""] --webHookUrl Set webhook url to receive scraper result as HTTP requests. For example to your own API [default: ""] --method Receive data to your webhook url as POST or GET request [choices: "GET", "POST"] [default: "POST"] --help Show help [boolean]
Examples: tiktok-scraper user USERNAME -d -n 100 --session sid_tt=dae32131231 tiktok-scraper trend -d -n 100 --session sid_tt=dae32131231 tiktok-scraper hashtag HASHTAG_NAME -d -n 100 --session sid_tt=dae32131231 tiktok-scraper music MUSIC_ID -d -n 50 --session sid_tt=dae32131231 tiktok-scraper video https://www.tiktok.com/@tiktok/video/6807491984882765062 -d tiktok-scraper history tiktok-scraper history -r user:bob tiktok-scraper history -r all tiktok-scraper from-file BATCH_FILE ASYNC_TASKS -d
- [Примеры терминала](https://github.com/drawrowfly/tiktok-scraper/tree/master/examples/CLI/Examples.md)
- [Управление историей загрузок](https://github.com/drawrowfly/tiktok-scraper/tree/master/examples/CLI/DownloadHistory.md)
- [Сбор и пакетная загрузка](https://github.com/drawrowfly/tiktok-scraper/tree/master/examples/CLI/BatchDownload.md)
### Пример выходного файла
## Docker
При использовании docker вы не сможете использовать --filepath и --historypath, но можете установить том (**путь на хосте, где будут сохранены все файлы**) с помощью -v
##### Сборка```sh
docker build . -t tiktok-scraper
Пример 1: Все файлы, включая файл истории, будут сохранены в каталоге ($pwd), откуда вы запускаете docker```sh docker run -v $(pwd):/usr/app/files tiktok-scraper user tiktok -d -n 5 -s
**Пример 2:**
Все файлы, включая файл истории, будут сохранены в /User/blah/downloads```sh
docker run -v /User/blah/downloads:/usr/app/files tiktok-scraper user tiktok -d -n 5 -s
.user(id, options) //Scrape posts from a specific user (Promise) .hashtag(id, options) //Scrape posts from hashtag section (Promise) .trend('', options) // Scrape posts from a trends section (Promise) .music(id, options) // Scrape posts by music id (Promise)
.userEvent(id, options) //Scrape posts from a specific user (Event) .hashtagEvent(id, options) //Scrape posts from hashtag section (Event) .trendEvent('', options) // Scrape posts from a trends section (Event) .musicEvent(id, options) // Scrape posts by music id (Event)
.getUserProfileInfo('USERNAME', options) // Get user profile information .getHashtagInfo('HASHTAG', options) // Get hashtag information .signUrl('URL', options) // Get signature for the request .getVideoMeta('WEB_VIDEO_URL', options) // Get video meta info, including video url without the watermark .getMusicInfo('https://www.tiktok.com/music/original-sound-6801885499343571718', options) // Get music metadata
### Параметры```javascript
const options = {
// Number of posts to scrape: {int default: 20}
number: 50,
// Scrape posts published since this date: { int default: 0}
since: 0,
// Set session: {string[] default: ['']}
// Authenticated session cookie value is required to scrape user/trending/music/hashtag feed
// You can put here any number of sessions, each request will select random session from the list
sessionList: ['sid_tt=21312213'],
// Set proxy {string[] | string default: ''}
// http proxy: 127.0.0.1:8080
// socks proxy: socks5://127.0.0.1:8080
// You can pass proxies as an array and scraper will randomly select a proxy from the array to execute the requests
proxy: '',
// Set to {true} to search by user id: {boolean default: false}
by_user_id: false,
// How many post should be downloaded asynchronously. Only if {download:true}: {int default: 5}
asyncDownload: 5,
// How many post should be scraped asynchronously: {int default: 3}
// Current option will be applied only with current types: music and hashtag
// With other types it is always 1 because every request response to the TikTok API is providing the "maxCursor" value
// that is required to send the next request
asyncScraping: 3,
// File path where all files will be saved: {string default: 'CURRENT_DIR'}
filepath: `CURRENT_DIR`,
// Custom file name for the output files: {string default: ''}
fileName: `CURRENT_DIR`,
// Output with information can be saved to a CSV or JSON files: {string default: 'na'}
// 'csv' to save in csv
// 'json' to save in json
// 'all' to save in json and csv
// 'na' to skip this step
filetype: `na`,
// Set custom headers: user-agent, cookie and etc
// NOTE: When you parse video feed or single video metadata then in return you will receive {headers} object
// that was used to extract the information and in order to access and download video through received {videoUrl} value you need to use same headers
headers: {
'user-agent': "BLAH",
referer: 'https://www.tiktok.com/',
cookie: `tt_webid_v2=68dssds`,
},
// Download video without the watermark: {boolean default: false}
// Set to true to download without the watermark
// This option will affect the execution speed
noWaterMark: false,
// Create link to HD video: {boolean default: false}
// This option will only work if {noWaterMark} is set to {true}
hdVideo: false,
// verifyFp is used to verify the request and avoid captcha
// When you are using proxy then there are high chances that the request will be
// blocked with captcha
// You can set your own verifyFp value or default(hardcoded) will be used
verifyFp: '',
// Switch main host to Tiktok test enpoint.
// When your requests are blocked by captcha you can try to use Tiktok test endpoints.
useTestEndpoints: false
};
Не забудьте проверить папку examples
const TikTokScraper = require('tiktok-scraper');
// User feed by username (async () => { try { const posts = await TikTokScraper.user('USERNAME', { number: 100, sessionList: ['sid_tt=58ba9e34431774703d3c34e60d584475;'] }); console.log(posts); } catch (error) { console.log(error); } })();
// User feed by user id
// Some TikTok user id's are larger then MAX_SAFE_INTEGER, you need to pass user id as a string
(async () => {
try {
const posts = await TikTokScraper.user(USER_ID, {
number: 100,
by_user_id: true,
sessionList: ['sid_tt=58ba9e34431774703d3c34e60d584475;']
});
console.log(posts);
} catch (error) {
console.log(error);
}
})();
// Trending feed (async () => { try { const posts = await TikTokScraper.trend('', { number: 100, sessionList: ['sid_tt=58ba9e34431774703d3c34e60d584475;'] }); console.log(posts); } catch (error) { console.log(error); } })();
// Hashtag feed (async () => { try { const posts = await TikTokScraper.hashtag('HASHTAG', { number: 100, sessionList: ['sid_tt=58ba9e34431774703d3c34e60d584475;'] }); console.log(posts); } catch (error) { console.log(error); } })();
// Get single user profile information: Number of followers and etc // input - USERNAME // options - not required (async () => { try { const user = await TikTokScraper.getUserProfileInfo('USERNAME', options); console.log(user); } catch (error) { console.log(error); } })();
// Get single hashtag information: Number of views and etc // input - HASHTAG NAME // options - not required (async () => { try { const hashtag = await TikTokScraper.getHashtagInfo('HASHTAG', options); console.log(hashtag); } catch (error) { console.log(error); } })();
// Get single video metadata // input - WEB_VIDEO_URL // For example: https://www.tiktok.com/@tiktok/video/6807491984882765062 // options - not required (async () => { try { const videoMeta = await TikTokScraper.getVideoMeta('https://www.tiktok.com/@tiktok/video/6807491984882765062', options); console.log(videoMeta); } catch (error) { console.log(error); } })();
### Событие```javascript
const TikTokScraper = require('tiktok-scraper');
const users = TikTokScraper.userEvent("tiktok", { number: 30 });
users.on('data', json => {
//data in JSON format
});
users.on('done', () => {
//completed
});
users.on('error', error => {
//error message
});
users.scrape();
const hashtag = TikTokScraper.hashtagEvent("summer", { number: 250, proxy: 'socks5://1.1.1.1:90' });
hashtag.on('data', json => {
//data in JSON format
});
hashtag.on('done', () => {
//completed
});
hashtag.on('error', error => {
//error message
});
hashtag.scrape();
НЕ ТРЕБУЕТСЯ
Очень распространенная проблема — когда TikTok блокирует ваш IP/ПРОКСИ. В таком случае можно попробовать установить сессию, и шансы на успех возрастут.
Получение сессии:
Установка сессии:
CLI:
sid_tt=521kkadkasdaskdj4j213j12j312;
sid_tt=521kkadkasdaskdj4j213j12j312;
sid_tt=521kkadkasdaskdj4j213j12j312;
sid_tt=521kkadkasdaskdj4j213j12j312;
В МОДУЛЕ вы можете установить сессию, задав значение опции sessionList. Например, sessionList:["sid_tt=521kkadkasdaskdj4j213j12j312;", "sid_tt=12312312312312;"]
Эта часть относится к использованию МОДУЛЯ (НЕ CLI)
Значение {videoUrl} привязано к значению cookie {tt_webid_v2}, которое может содержать любое значение
Когда вы извлекаете видео из ленты пользователя, хэштега, музыки, трендов или отдельного видео, то в ответе помимо метаданных видео вы получите объект headers, содержащий параметры, использованные для извлечения данных. Важный момент: чтобы получить доступ к видео / скачать его по значению {videoUrl}, необходимо использовать те же значения {headers}.```json headers: { "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36", "referer": "https://www.tiktok.com/", "cookie": "tt_webid_v2=689854141086886123" },
#### Метод 2: пользовательские заголовки
Вы можете передавать свои собственные заголовки с помощью **{options}**.```javascript
const headers = {
"user-agent": "BOB",
"referer": "https://www.tiktok.com/",
"cookie": "tt_webid_v2=BOB"
}
getVideoMeta('WEB_VIDEO_URL', {headers})
user('WEB_VIDEO_URL', {headers})
hashtag('WEB_VIDEO_URL', {headers})
trend('WEB_VIDEO_URL', {headers})
music('WEB_VIDEO_URL', {headers})
// And after you can access video through {videoUrl} value by using same custom headers
Пример вывода для методов: user, hashtag, trend, music, userEvent, hashtagEvent, musicEvent, trendEvent```javascript { headers: { 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36', referer: 'https://www.tiktok.com/', cookie: 'tt_webid_v2=689854141086886123' }, collector:[{ id: 'VIDEO_ID', text: 'CAPTION', createTime: '1583870600', authorMeta:{ id: 'USER ID', name: 'USERNAME', following: 195, fans: 43500, heart: '1093998', video: 3, digg: 95, verified: false, private: false, signature: 'USER BIO', avatar:'AVATAR_URL' }, musicMeta:{ musicId: '6808098113188120838', musicName: 'blah blah', musicAuthor: 'blah', musicOriginal: true, playUrl: 'SOUND/MUSIC_URL', }, covers:{ default: 'COVER_URL', origin: 'COVER_URL', dynamic: 'COVER_URL' }, imageUrl:'IMAGE_URL', videoUrl:'VIDEO_URL', videoUrlNoWaterMark:'VIDEO_URL_WITHOUT_THE_WATERMARK', videoMeta: { width: 480, height: 864, ratio: 14, duration: 14 }, diggCount: 2104, shareCount: 1, playCount: 9007, commentCount: 50, mentions: ['@bob', '@sam', '@bob_again', '@and_sam_again'], hashtags: [{ id: '69573911', name: 'PlayWithLife', title: 'HASHTAG_TITLE', cover: [Array] }...], downloaded: true }...], //If {filetype} and {download} options are enbabled then: zip: '/{CURRENT_PATH}/user_1552963581094.zip', json: '/{CURRENT_PATH}/user_1552963581094.json', csv: '/{CURRENT_PATH}/user_1552963581094.csv' }
##### getUserProfileInfo```javascript
{
secUid: 'MS4wLjABAAAAv7iSuuXDJGDvJkmH_vz1qkDZYo1apxgzaxdBSeIuPiM',
userId: '107955',
isSecret: false,
uniqueId: 'tiktok',
nickName: 'TikTok',
signature: 'Make Your Day',
covers: ['COVER_URL'],
coversMedium: ['COVER_URL'],
following: 490,
fans: 38040567,
heart: '211522962',
video: 93,
verified: true,
digg: 29,
}
{ challengeId: '4231', challengeName: 'love', text: '', covers: [], coversMedium: [], posts: 66904972, views: '194557706433', isCommerce: false, splitTitle: '' }
##### getVideoMeta```javascript
{
headers: {
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36',
referer: 'https://www.tiktok.com/',
cookie: 'tt_webid_v2=689854141086886123'
},
collector:[{
id: '6807491984882765062',
text: 'We’re kicking off the #happyathome live stream series today at 5pm PT!',
createTime: '1584992742',
authorMeta: { id: '6812221792183403526', name: 'blah' },
musicMeta:{
musicId: '6822233276137213677',
musicName: 'blah',
musicAuthor: 'blah'
},
imageUrl: 'IMAGE_URL',
videoUrl: 'VIDEO_URL',
videoUrlNoWaterMark: 'VIDEO_URL_WITHOUT_THE_WATERMARK',
videoMeta: { width: 480, height: 864, ratio: 14, duration: 14 },
covers:{
default: 'COVER_URL',
origin: 'COVER_URL'
},
diggCount: 49292,
shareCount: 339,
playCount: 614678,
commentCount: 4023,
downloaded: false,
hashtags: [],
}]
}
{ music: { id: '6882925279036066566', title: 'doja x calabria', playUrl: 'dfdfdfdf', coverThumb: 'dfdfdf', coverMedium: 'dfdfdf', coverLarge: 'fdfdf', authorName: 'bryce', original: true, playToken: 'ffdfdf', keyToken: 'dfdfdfd', audioURLWithcookie: false, private: false, duration: 46, album: '', }, author: { id: '6835300004094166021', uniqueId: 'mashupsbybryce', nickname: 'bryce', avatarThumb: 'dfdfd', avatarMedium: 'dfdfdf', avatarLarger: 'dfdfdf', signature: 'hi ily :)\n70k sounds cool tbh\n👇follow my soundcloud & insta👇', verified: false, secUid: 'MS4wLjABAAAA1_5bjLAamayD4rv3q49qJGa_7dZ5jzExTO0ozOybqIwwhw5TAg_iM25lkO94DM3K', secret: false, ftc: false, relation: 0, openFavorite: false, commentSetting: 0, duetSetting: 0, stitchSetting: 0, privateAccount: false, }, stats: { videoCount: 361700 }, shareMeta: { title: 'bryceyouloser | ♬ doja x calabria | on TikTok', desc: '361.0k videos - Watch awesome short ' + 'videos created with ♬ doja x calabria', }, };
<a href="https://www.buymeacoffee.com/Usom2qC" target="_blank"><img src="https://assets.kitploit.com/production/public/readmes/5141/c20d7d7728f977412007ddec6509758f76c0727e972aea2c4c18ead6c69ff2af.png" alt="Buy Me A Coffee" style="height: 41px !important;width: 174px !important;box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;-webkit-box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;" ></a>
---
Лицензия
---
**MIT**
**Бесплатное программное обеспечение**