ultrafunkamsterdam/nodriverSuccessor of Undetected-Chromedriver. Providing a blazing fast framework for web automation, webscraping, bots and any other creative ideas which are normally hindered by annoying anti bot systems like Captcha / CloudFlare / Imperva / hCaptcha
Direct communication provides even better resistance against web applicatinon firewalls (WAF’s), while performance gets a massive boost. This module is, contrary to undetected-chromedriver, fully asynchronous.
What makes this package different from other known packages, is the optimization to stay undetected for most anti-bot solutions.
Another focus point is usability and quick prototyping, so expect a lot to work -as is- ,
with most method parameters having best practice defaults.
Using 1 or 2 lines, this is up and running, providing best practice config
by default. It cleans up created files (profile) afterwards.
known to work with
While usability and convenience is important. It’s also easy to fully customizable everything using the entire array of CDP domains, methods and events available.
No chromedriver binary or Selenium dependency
Up and running in 1 line of code*
uses fresh profile on each run, cleans up on exit
save and load cookies to file to not repeat tedious login steps
tab.find("sometext")
tab.find_all("sometext")
tab.select("a[class*=something]")
tab.select_all("a[href] > div > img")
smart and performant element lookup, by selector or text, including iframe content.
this could also be used as wait condition for a element to appear, since it will retry
for the duration of until found. so an await tab.select('body') could be used
as an indicator whether a page is loaded.
the find method searches by text, but will not naively return the first
matching element, but will match candidates by closest matching text length (shortest wins),
this makes lookups like tab.find('accept all') return the actual cookie button instead of
a script in the headers
can connect to a running chrome debug session
descriptive __repr__ for elements, which represent the element as html
utility function to convert a running undetected_chromedriver.Chrome instance to a nodriver.Browser instance and contintue from there
packed with helpers and utility methods for most used and important operations
Parts are rewritten to use flat connections in the protocol.
Why?
- iframes are included in most operations.
- tab got a new method: await tab.get_frames()
which will return Iframes that are inspectable.
- find() will include iframes, so you can even search for "verify you are human" and
click the verification checkbox in js challenges.
Since this required quite some rewriting, please test thoroughly, especially if you run large projects.
tab.xpath(selector, timeout=2.5)find nodes using xpath selector! see tab xpath in the api docs
tab.cf_verify()finds the checkbox and click it successfully this only works when NOT in expert mode. currently built-in english only requires opencv-python package to be installed
tab.bypass_insecure_connection_warning()convenience method, for insecure page warning. for example when a certificate is invalid.
tab.open_external_debugger()lets you inspect the tab without breaking your connection
tab.get_local_storage()get localstorage content
tab.set_local_storage(dict)set localstorage content
tab.add_handler(someEvent, callback)callback may accept a single argument (event), or 2 arguments (event, tab).
start(expert=True)does some hacking for more experienced users. It disables web security and origin-trials, as well as ensures shadow-roots are always open. This makes you more detectable though!
you need chrome (or some chromium based browser) installed preferably in the default location on the machine where you use this package.
when running on a headless machine, like AWS or any other environment where no display is present, it's best to use some Xvfb tool, to emulate a screen. alternatively this package can be used in headless mode.
pip install nodriver
pip install -U nodriver
The aim of this project (just like undetected-chromedriver, somewhere long ago) is to keep it short and simple, so you can quickly open an editor or interactive session, type or paste a few lines and off you go.
import nodriver as uc
async def main():
browser = await uc.start()
page = await browser.get('https://www.nowsecure.nl')
... further code ...
if __name__ == '__main__':
# since asyncio.run never worked (for me)
uc.loop().run_until_complete(main())
I’ll leave out the async boilerplate here
from nodriver import *
browser = await start(
headless=False,
user_data_dir="/path/to/existing/profile", # by specifying it, it won't be automatically cleaned up when finished
browser_executable_path="/path/to/some/other/browser",
browser_args=['--some-browser-arg=true', '--some-other-option'],
lang="en-US" # this could set iso-language-code in navigator, not recommended to change
)
tab = await browser.get('https://somewebsite.com')
I’ll leave out the async boilerplate here
from nodriver import *
config = Config()
config.headless = False
config.user_data_dir="/path/to/existing/profile", # by specifying it, it won't be automatically cleaned up when finished
config.browser_executable_path="/path/to/some/other/browser",
config.browser_args=['--some-browser-arg=true', '--some-other-option'],
config.lang="en-US" # this could set iso-language-code in navigator, not recommended to change
)
import nodriver
async def main():
browser = await nodriver.start()
page = await browser.get('https://www.nowsecure.nl')
await page.save_screenshot()
await page.get_content()
await page.scroll_down(150)
elems = await page.select_all('*[src]')
for elem in elems:
await elem.flash()
page2 = await browser.get('https://twitter.com', new_tab=True)
page3 = await browser.get('https://github.com/ultrafunkamsterdam/nodriver', new_window=True)
for p in (page, page2, page3):
await p.bring_to_front()
await p.scroll_down(200)
await p # wait for events to be processed
await p.reload()
if p != page3:
await p.close()
if __name__ == '__main__':
# since asyncio.run never worked (for me)
uc.loop().run_until_complete(main())
automating Twitter/X account creation
A more concrete example, which can be found in the ./example/ folder,
shows a script to create a twitter account
```python
import random
import string
import logging