GyoiThon: Next generation penetration test tool

Japanese page is here.

Presentation

• January 25th,2018:JANOG41
• March 23th,2018:Black Hat ASIA 2018 Arsenal
• August 12th,2018:DEFCON26 DemoLabs
• October 24th,2018:OWS in CSS2018
• November 3rd,2018:AV TOKYO 2018 HIVE
• December 22-23th,2018:SECCON YOROZU 2018
• March 28th,2019:Black Hat ASIA 2019 Arsenal

Documents

• Installation
• Usage
• Tips
• Demonstration (Youtube)

Slack

• https://gyoithon.slack.com
  Let’s join GyoiThon Slack!!

New function!!

The new GyoiThon (version 0.0.4) can list up your subdomain facing on the internet. And if the subdomain is published as a Web service, then GyoiThon executes a health check that a non-destructive vulnerability assessment.

• ex) Listing up your subdomain.
  First, you have to prepare the domain_list.csv is following:
  "Domain Name" mbsd.jp
  And you execute following command.
  root@kali:~/GyoiThon# python3 gyoithon.py -i --domain_list
  As a result, you get a list of sundomains associated with the specified domain.
  |Index|Domain|Sub-Domain|IP Address|Access Status (http)|Location (http)|Access Status (https)|Location (https)|Whois records|
  |:–:|:–:|:–:|:–:|:–:|:–:|:–:|:–:|:–:|
  | 1 | mbsd.jp | mbsd.jp | [‘40.115.251.148’] | 301 | https://www.mbsd.jp/ | 301 | https://www.mbsd.jp/ | *** |
  | 2 | mbsd.jp | www.mbsd.jp | [‘40.115.251.148’] | 301 | https://www.mbsd.jp/ | 200 | - | - |
  | 3 | mbsd.jp | www2.mbsd.jp | [‘40.115.251.148’] | 301 | https://www.mbsd.jp/ | 200 | - | - |

• ex) Listing up your subdomain and executing health check.
  GyoiThon executes a list up your subdomain and if the subdomain is published as a Web service (with port 80 or 443 open), then GyoiThon execute a health check that a non-destructive vulnerability assessment.

root@kali:~/GyoiThon# python3 gyoithon.py -i --domain_list --through_health_check --safety

As a result, you get a list of subdomains and assessment report.
|Index|Domain|Sub-Domain|IP Address|Access Status (http)|Location (http)|Access Status (https)|Location (https)|Whois records|Assessment results|
|:–:|:–:|:–:|:–:|:–:|:–:|:–:|:–:|:–:|:–:|
| 1 | mbsd.jp | mbsd.jp | [‘40.115.251.148’] | 301 | https://www.mbsd.jp/ | 301 | https://www.mbsd.jp/ | *** | *** |
| 2 | mbsd.jp | www.mbsd.jp | [‘40.115.251.148’] | 301 | https://www.mbsd.jp/ | 200 | - | - | *** |
| 3 | mbsd.jp | www2.mbsd.jp | [‘40.115.251.148’] | 301 | https://www.mbsd.jp/ | 200 | - | - | *** |

Overview

GyoiThon is Intelligence Gathering tool for Web Server.

GyoiThon execute remote access to target Web server and identifies product operated on the server such as CMS, Web server software, Framework, Programming Language etc,. And, it can execute exploit modules to identified products using Metasploit. GyoiThon fully automatically execute above action.

GyoiThon’s main features are following.

• Remote access/Fully automatic
  GyoiThon can fully automatically gather the information of target Web server using only remote access. You only execute GyoiThon once for your operation.

• Non-destructive test
  GyoiThon can gather information of target Web server using only normally access.
  But, when you use a part of option, GyoiThon execute abnormally access such as sending exploit modules.

• Gathering various information
  GyoiThon has various intelligence gathering engines such as Web crawler, Google Custom Search API, Censys, explorer of default contents, examination of cloud services etc,. By analyze gathered information using strings pattern matching and machine learning, GyoiThon can identify product/version/CVE number operated on the target web server, unnecceary html comments/debug messages, login page etc,.

• Examination of real vulnerability
  GyoiThon can execute exploit modules to identified products using Metasploit.
  As a result, it can examine real vulnerability of target web server.

Installation

1. git clone GyoiThon’s repository.

root@kali:~# git clone https://github.com/gyoisamurai/GyoiThon.git

2. Get python3-pip.

root@kali:~# apt-get update
root@kali:~# apt-get install python3-pip

3. install required python packages.

root@kali:~# cd GyoiThon
root@kali:~/GyoiThon# pip3 install -r requirements.txt
root@kali:~/GyoiThon# apt install python3-tk

4. Edit config.ini of GyoiThon.
   You have to edit your config.ini.
   More information is Usage.

Usage

By using default mode without option and combination of several options, GyoiThon can gather various information of target web server.

usage:
    .\gyoithon.py [-s] [-m] [-g] [-e] [-c] [-p] [-l --log_path=<path>] [--no-update-vulndb]
    .\gyoithon.py [-d --category=<category> --vendor=<vendor> --package=<package>]
    .\gyoithon.py [-i]
    .\gyoithon.py -h | --help
options:
    -s   Optional : Examine cloud service.
    -m   Optional : Analyze HTTP response for identify product/version using Machine Learning.
    -g   Optional : Google Custom Search for identify product/version.
    -e   Optional : Explore default path of product.
    -c   Optional : Discover open ports and wrong ssl server certification using Censys.
    -p   Optional : Execute exploit module using Metasploit.
    -l   Optional : Analyze log based HTTP response for identify product/version.
    -d   Optional : Development of signature and train data.
    -i   Optional : Explore relevant FQDN with the target FQDN.
    -h --help     Show this help message and exit.

Preparation.

1. Edit target file host.txt.
   You have to write target web server to the host.txt.
   Writting format is protocol FQDN(or IP address) Port Crawling_root_path.

• Example.

https gyoithon.example.com 443 /

If you want to indicate multiple target information, you have to write below.

https gyoithon.example.com 443 /
http 192.168.220.129 80 /vicnum/
https www.example.com 443 /catalog/

2. Edit configuration file config.ini.
   Parameters to be changed by the user are defined in the setting file config.ini.
   If you want to change parameters, edit config.ini.
   Detail of config.ini is here.

Execution of GyoiThon.

1. Default mode.

root@kali:~/GyoiThon# python3 gyoithon.py

The default mode gathers following minimum information.

1. Gathering of HTTP responses by Web crawling.
2. Identification of product/version using string pattern matching.
3. Examination of CVE number (from NVD) for identified products.
4. Examination of unneccesary HTML/JavaScript comments.
5. Examination of unneccesary debug messages.
6. Examination of login pages.

• Crawling setting
  GyoiThon uses Scrapy that Python’s library.
  By change the parameters in config.ini, you can change setting of Scrapy.

• Examination speed setting
  The examination number and HTTP response size greatly affect examination times.
  By change the parameters in config.ini, you can adjust examination speed.

2. Examination of cloud services mode.

root@kali:~/GyoiThon# python3 gyoithon.py -s

By add -s option, GyoiThon identifies target web server uses cloud service or not in addition to default mode.
Before execution, you must change the below parameter of config.ini.

This parameter is source URL of Azure Datacenter IP range. This URL is changed a few per day. So, you must get the latest URL from link “click here to download manually” of page “Microsoft Azure Datacenter IP Ranges” and set it to above parameter before execute GyoiThon.

3. Machine Learning analysis mode.

root@kali:~/GyoiThon# python3 gyoithon.py -m

By add -m option, GyoiThon identifies products/version using Machine Learning (Naive Bayes) in addition to default mode.

4. Google Hacking mode.

root@kali:~/GyoiThon# python3 gyoithon.py -g

By add -g option, GyoiThon identifies products/version using Google Custom Search API in addition to default mode. Before execution, you must set API key and Search engine ID to the below parameters.

5. Exploration of default contents mode.

root@kali:~/GyoiThon# python3 gyoithon.py -e

By add -e option, GyoiThon explores the default contents of products such as CMS, Web server software in addition to default mode.
By change the parameters in config.ini, you can change setting of exploration.

6. Censys cooperation mode.

root@kali:~/GyoiThon# python3 gyoithon.py -c

By add -c option, GyoiThon examines open port number and server certification using Censys.
Before execution, you must set API key and Secret key to the below parameters.

7. Metasploit cooperation mode.

root@kali:~/GyoiThon# python3 gyoithon.py -p

By add -p option, GyoiThon examines real vulnerabilities such as DoS and backdoor using Metasploit in addition to default mode.
Before execution, you must launch RPC server of Metasploit and set below parameters in config.ini.

8. Stored logs based analysis mode.

root@kali:~/GyoiThon# python3 gyoithon.py -l --log_path="Full path of stored logs"

By add -l option, GyoiThon executes various examination using stored HTTP responses without web crawling.

This mode assumes the web application that GyoiThon cannot execute web crawling.
GyoiThon can execute various examination similar web crawling of default mode using stored HTTP responses gathered by local proxy tool.

9. Combination of multiple options.

Combination of “Examination of cloud services mode” and “Machine Learning analysis mode”.

root@kali:~/GyoiThon# python3 gyoithon.py -s -m

Combination of “Examination of cloud services mode” and “Google Hacking mode”.

root@kali:~/GyoiThon# python3 gyoithon.py -s -g

Combination of “Examination of cloud services mode”, “Machine Learning analysis mode” and “Google Hacking mode”.

root@kali:~/GyoiThon# python3 gyoithon.py -s -m -g

All option.

root@kali:~/GyoiThon# python3 gyoithon.py -s -m -g -e -c -p -l --log_path="Full path of stored logs"

10. Generating signature/train data mode.

root@kali:~/GyoiThon# python3 gyoithon.py -d --category=CMS --vendor=joomla! [email protected]@_origin.tar.zip

By add -d option, GyoiThon generates signatures/train data for detecting Web products operated on the target Web Server.
The options is following.

Check report.

After finished execution of GyoiThon, reports of each target are generated to the following path.

root@kali:~/GyoiThon/report# ls
gyoithon_report_192.168.220.129_80_1082018338.csv
gyoithon_report_192.168.220.129_80_bodgeit.csv
gyoithon_report_192.168.220.129_80_cyclone.csv
gyoithon_report_192.168.220.129_80_vicnum.csv
gyoithon_report_192.168.220.129_80_WackoPicko.csv
gyoithon_censys_report_www.gyoithon.example.com_443_test.csv

GyoiThon generates following two types report.

• gyoithon_report_target FQDN(or IP address)_Port number_Root Path.csv.
  This is main report that mainly including product name, version, cve etc,.
  Report format is gyoithon_report_target FQDN(or IP address)_Port number_Root Path.csv.
  Each column’s detail is following.

• gyoithon_censys_report_target FQDN(or IP address)_Port number_Root Path.csv.
  This is search result report using Censys that including open ports, certification information etc,.
  Report format is gyoithon_censys_report_target FQDN(or IP address)_Port number_Root Path.csv.
  Each column’s detail is following.

Tips

1. How to manually add new signature (string matching patterns).

signatures path includes below files.

root@kali:~/GyoiThon/signatures/ ls
signature_product.txt
signature_default_content.txt
signature_search_query.txt
signature_comment.txt
signature_error.txt
signature_page_type_from_url.txt

signature_product.txt

This is string matching patterns for identification of product in default mode.
If you want to add new string matching pattern, you have to write it such following format.

Format: field1@field2@field3@field4@field5

If you don’t need optional field, you must set * to this field.

• Example

CMS@wordpress@wordpress@*@.*(WordPress ([0-9]+[\.0-9]*[\.0-9]*)).*
CMS@drupal@[email protected]@.*(X-Generator: Drupal 8).*

signature_default_content.txt

This is string matching patterns for identification of product in Exploration of default contents mode.
If you want to add new string matching pattern, you have to write it such following format.

Format: field1@field2@field3@field4@field5@field6@field7@field8

If you don’t need optional field, you must set * to this field.

• Example

Web@apache@http_server@*@/server-status@*@Version:.*(Apache/([0-9]+[\.0-9]*[\.0-9]*))@0
CMS@sixapart@movabletype@*@/readme.html@.*(Movable Type).*@(v=([0-9]+[\.0-9]*[\.0-9]*))@0

signature_search_query.txt

This is Google Custom Search query for identification of product in Google Hacking mode.
If you want to add new query, you have to write it such following format.

Format: field1@field2@field3@field4@field5@field6@field7@field8

If you don’t need optional field, you must set * to this field.

• Example

Web@apache@http_server@*@inurl:/server-status@*@Version:.*(Apache/([0-9]+[\.0-9]*[\.0-9]*))@0
CMS@sixapart@movabletype@*@inurl:/readme.html@.*(Movable Type).*@(v=([0-9]+[\.0-9]*[\.0-9]*))@0
*@*@*@*@filetype:bak@*@*@0

signature_comment.txt

This is string matching patterns for identification of unnecessary comments in default mode.
If you want to add new string matching pattern, you have to write it such following format.

Format: field1

• Example

(user\s*=|[\"']user[\"']\s*:|user_id\s*=|[\"']user_id[\"']\s*:|id\s*=|[\"']id[\"']\s*:)
(select\s+[\s\r\n\w\d,\"']*\s+from)

signature_error.txt

This is string matching patterns for identification of unnecessary debug message in default mode.
If you want to add new string matching pattern, you have to write it such following format.

Format: field1

• Example

(ORA-[0-9a-zA-Z\.])
(fail|error|notice|parse|warning|fatal)[^\n]*line[^\n]*[0-9]+

signature_page_type_from_url.txt

This is string matching patterns for URL based identification of page type in default mode.
If you want to add new string matching pattern, you have to write it such following format.

Format: field1@field2

• Example

Login@.*(login|log_in|logon|log_on|signin|sign_in).*

2. How to manually add learning data.

modules/train_data/ path includes two train data for Machine Learning.

root@kali:~/GyoiThon/modules/train_data/ ls
train_cms_in.txt
train_page_type.txt

train_cms_in.txt

This is train data for Machine Learning analysis in Machine Learning mode.
If you want to add new train data, you have to write it such following format.

Format: field1@field2@field3@field4

If you don’t need optional field, you must set * to this field.

• Example

joomla@joomla\!@*@(Set-Cookie: [a-z0-9]{32}=.*);
joomla@joomla\!@*@(Set-Cookie: .*=[a-z0-9]{26,32});
heartcore@heartcore@*@(Set-Cookie:.*=[A-Z0-9]{32});.*
heartcore@heartcore@*@(<meta name=["']author["'] content=["']{2}).*

train_page_type.txt

This is train data for identifying page type usin Machine Learning in default mode.
If you want to add new train data, you have to write it such following format.

Format: field1@field2

• Example

Login@.*(<input.*type=[\"']text[\"'].*name=[\"']user|uid|username|user_name|name[\"']).*>
Login@.*(<input.*type=[\"']password[\"']).*>

3. How to change “Exploit module’s option”.

When GyoiThon exploits, it uses default value of Exploit module options.
If you want to change option values, please input any value to "user_specify" in exploit_tree.json as following.

root@kali:~/GyoiThon/modules/data/ ls
exploit_tree.json
root@kali:~/GyoiThon/modules/data/ vim exploit_tree.json

...snip...

"unix/webapp/joomla_media_upload_exec": {
    "targets": {
        "0": [
            "generic/custom",
            "generic/shell_bind_tcp",
            "generic/shell_reverse_tcp",

...snip...

        "TARGETURI": {
            "type": "string",
            "required": true,
            "advanced": false,
            "evasion": false,
            "desc": "The base path to Joomla",
            "default": "/joomla",
            "user_specify": "/my_original_dir/"
        },

Above example is to change value of TARGETURI option in exploit module “exploit/unix/webapp/joomla_media_upload_exec” to “/my_original_dir/” from “/joomla”.

4. How to access via proxy server.

GyoiThon can access to target server via proxy server.
If you want to use proxy server, please input proxy server information in config.ini.

Operation check environment

• Kali Linux 2018.2 (for Metasploit)
• CPU: Intel(R) Core(TM) i5-5200U 2.20GHz
• Memory: 8.0GB
• Metasploit Framework 4.16.48-dev
• Python 3.6.6

Licence

Apache License 2.0

Contact us

[email protected]

• Masafumi Masuya
  https://twitter.com/gyoizamurai
• Toshitsugu Yoneyama
  https://twitter.com/yoneyoneyo
• Isao Takaesu
  https://twitter.com/bbr_bbq