Project Overview – Automation Using Cron Jobs, Python Scripts, and MMDB

Current Project Status

Note: The project is currently partially functional. At this stage, users can register an account or log in using the following test credentials:

Username: test
Password: 1

Development is ongoing, and new functionalities are actively being added to enhance the platform.

Introduction

This project describes an automated solution for managing and validating proxy lists. It uses Cron Jobs, Python scripts, public-private key-based SSH authentication, and the MMDB (MaxMind Database) for geolocation data. The goal is to demonstrate a fully automated process for downloading, validating, and processing proxy data, then storing it in a database.

Automated Workflow

1. Initiation via Cron Jobs

Frequency: Every 3 hours, a Cron Job runs a command to start the process.
Command: ssh root@87.120.114.174 "python3 /root/Bubu/main.py"
Authentication: The server connection uses SSH Key Authentication for a secure, password-free login.

2. The Main Script – `main.py`

The main.py script orchestrates the execution of other scripts, ensuring each has enough time to complete:

get_proxy.py – 300 seconds (5 minutes)
check_proxy.py – 1800 seconds (30 minutes)
json_vorte.py – 300 seconds (5 minutes)
transfer_proxy.py – 300 seconds (5 minutes)

Process Details

1. `get_proxy.py` – Downloading Proxy Lists

Downloads proxy lists from public sources and saves them locally.

SOCKS5: source_link
SOCKS4: source_link
HTTP: source_link
HTTPS: source_link

2. `check_proxy.py` – Validating Proxies

Verifies the functionality of downloaded proxies. While designed for demonstration, the script ensures enough proxies are functional for testing purposes.

3. `json_vorte.py` – Processing and Enriching Data

Adds metadata to each proxy using:

The MMDB (MaxMind Database) for geolocation (country, city, ASN).
Supplementary file ASN-INFO.txt for organization names.

Output is saved in a JSON format:

{
  "type": "SOCKS5",
  "country": "South Korea",
  "city": "Seoul",
  "ip": "8.213.129.15",
  "port": "9992",
  "asn": "45102",
  "organization": "Alibaba US Technology"
}

4. `transfer_proxy.py` – Data Transfer

Transfers the generated JSON file to the cPanel server, where it is integrated into the SQL database.

Authentication: Secure SSH connection using key pairs.

cPanel Workflow

1. Importing Data

Cron Job Command: curl -s https://vorte.eu/proxies/cached_proxies.php > /dev/null
Function: Reads the JSON file and imports data into the SQL table proxies.

2. Random Proxy Display

Cron Job Command: curl -s https://vorte.eu/proxies/update_proxies.php > /dev/null
Function: Selects 40 random proxies hourly and stores them in the cached_proxies table for display.

Features for Registered Users

View the complete list of proxies with advanced filtering options (type, country, city, ASN, organization).
Select and download up to 30 proxies daily for free.
Use the shopping cart functionality to manage and download selected proxies.
Access "My Proxy List" to view downloaded proxies without masked IPs.