关于web-traffic-generator
web-traffic-generator is a powerful HTTP and HTTPs traffic obfuscation tool, which is developed based on pure Python, which can help researchers add noise to HTTP or HTTPs network traffic to achieve the purpose of traffic obfuscation.
Essentially, web-traffic-generator is a network noise generation tool that can be used for incident response or cyber defense tasks on a red or blue team. web-traffic-generator has been tested on Ubuntu 14.04 & 16.04 and theoretically supports use on any operating system with a Python environment installed.
How the tool works
First, we need to specify some configuration information at the beginning of the script:
1. MAX_DEPTH = 10, MIN_DEPTH = 5: Starting from each root URL (e.g., www.yahoo.com), the tool will try to access a random depth of node addresses between the MIN_DEPTH and the MAX_DEPTH;
The time interval between each HTTP GET request is randomly selected based on the following variables:
1. MIN_WAIT = 5: The interval between requests is at least 5 seconds, if this time is too short, it may be blocked by the target web server.
2. MAX_WAIT = 10: The maximum interval between requests is 10 seconds.
3. DEBUG = False: Set the debugging mode, set it to True, turn on the Verbose mode, and the tool will print out the detailed output and debugging information in real time;
4. ROOT_URLS = [url1,url2,url3]: a list of root URL addresses, in which the tool will randomly select;
5、blacklist = [".gif", "intent/tweet", "badlink", etc...] : a blacklist of strings that need to be detected for each link, and if the link contains any string in this list, the link will be discarded;
6、userAgent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) .......': sets the user agent (headless browser) passed to the web server, which we can set as the default, or modify it as needed;
Tool environment
当前版本的web-traffic-generator同时兼容Python 2.7和Python 3.x环境。
Tool dependencies
The tool only needs to be run using the requests library, and the installation command is as follows:
sudo pip install requests Tool installation
Researchers can directly clone the source code of the project locally with the following command:
git clone https://github.com/ReconInfoSec/web-traffic-generator.git Tool use
First, we need to create a configuration file:
cp config.py.template config.py 然后直接运行web-traffic-generator即可：
python gen.py Debugging information
If we set the Debug variable in the config.py configuration file to True, we will see the Verbose output as follows:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Traffic generator started
Diving between 3 and 10 links deep into 489 different root URLs,
Waiting between 5 and 10 seconds between requests.
This script will run indefinitely. Ctrl+C to stop.
Randomly selecting one of 489 URLs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Recursively browsing [https://arstechnica.com] ~~~ [depth = 7]
Requesting page...
Page size: 77.6KB
Data meter: 77.6KB
Good requests: 1
Bad reqeusts: 0
Scraping page for links
Found 171 valid links
Pausing for 7 seconds...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Recursively browsing [https://arstechnica.com/author/jon-brodkin/] ~~~ [depth = 6]
Requesting page...
Page size: 75.7KB
Data meter: 153.3KB
Good requests: 2
Bad reqeusts: 0
Scraping page for links
Found 168 valid links
Pausing for 9 seconds...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Recursively browsing [https://arstechnica.com/information-technology/2020/01/directv-races-to-decommission-broken-boeing-satellite-before-it-explodes/] ~~~ [depth = 5]
Requesting page...
Page size: 43.8KB
Data meter: 197.1KB
Good requests: 3
Bad reqeusts: 0
Scraping page for links
Found 32 valid links
Pausing for 8 seconds...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Recursively browsing [https://www.facebook.com/sharer.php?u=https%3A%2F%2Farstechnica.com%2F%3Fpost_type%3Dpost%26p%3D1647915] ~~~ [depth = 4]
Requesting page...
Page size: 64.2KB
Data meter: 261.2KB
Good requests: 4
Bad reqeusts: 0
Scraping page for links
Found 0 valid links
Stopping and blacklisting: no links The last URL in the above output throws an error message because we added it to the config.blacklist array.
Screenshot of the tool running
Demo of tool usage
Demo video: https://asciinema.org/a/304683
License Agreement
The development and release of this project is under an open source license.
Project address
web-traffic-generator：https://github.com/ReconInfoSec/web-traffic-generator