urless

About - v2.7

This is a tool used to de-clutter a list of URLs. As a starting point, I took the amazing tool uro by Somdev Sangwan. But I wanted to change a few things, make some improvements (like deal with GUIDs) and make it more customizable.

Installation

urless supports Python 3.

Install urless in default (global) python environment.

pip install urless

OR

pip install git+https://github.com/xnl-h4ck3r/urless.git -v

You can upgrade with

pip install --upgrade urless

pipx

Quick setup in isolated python environment using pipx

pipx install git+https://github.com/xnl-h4ck3r/urless.git

Usage

Argument	Long Argument	Description
-i	--input	A file of URLs to de-clutter.
-o	--output	The output file that will contain the de-cluttered list of URLs (default: output.txt). If piped to another program, output will be written to STDOUT instead.
-fk	--filter-keywords	A comma separated list of keywords to exclude links (if there no parameters). This will override the FILTER_KEYWORDS list specified in config.yml
-fe	--filter-extensions	A comma separated list of file extensions to exclude. This will override the FILTER_EXTENSIONS list specified in config.yml
-rp	--remove-params	A comma separated list of case senistive parameters to remove from ALL URLs. This will override the REMOVE_PARAMS list specified in config.yml. This can be useful to remove cache buster parameters for example.**
-ks	--keep-slash	A trailing slash at the end of a URL in input will not be removed. Therefore there may be identical URLs output, one with and one without a trailing slash.
-khw	--keep-human-written	By default, any URL with a path part that contains more than 3 dashes (-) are removed because it is assumed to be human written content (e.g. blog post), and not interesting. Passing this argument will keep them in the output.
-kym	--keep-yyyymm	By default, any URL with a path containing /YYYY/MM (where YYYY is a year and MM month) are removed because it is assumed to be blog/news content, and not interesting. Passing this argument will keep them in the output.
-rcid	--regex-custom-id	USE WITH CAUTION! Regex for a Custom ID that your target uses. Ensure the value is passed in quotes. See the section below for more details on this.
-iq	--ignore-querystring	Remove the query string (including URL fragments #) so output is unique paths only.
-fnp	--fragment-not-param	Don't treat URL fragments # in the same way as parameters, e.g. if a link has a filter keyword and a fragment (or param) the link is usually kept, but if this argument is passed and a link has a filter word and fragment, the link will be removed. Also, if this arg is passed and -iq / --ignore-querystring is used, the fragment will NOT be removed from links if no query string is in the link.
-lang	--language	If passed and there are multiple URLs with different language codes as a part of the path, only one version of the URL will be output. The codes are specified in the LANGUAGE section of config.yml.
-c	--config	Path to the YML config file. If not passed, it looks for file config.yml in the default config directory, e.g. ~/.config/urless/.
-dp	--disregard-params	There is certain filtering that is not done if the URLs have parameters, because by default we want to see all possible parameters. If this argument is passed, then the filtering will be done, regardless of the existence of any parameters.