Dexscreener Scraper

Overview

Dexscreener Scraper is a Node.js application that fetches the latest token profiles from the Dexscreener API, scrapes their associated websites, and saves relevant information such as descriptions, header images, logos, and contracts to a local directory structure.

Features

• Fetches the latest token profiles from the Dexscreener API.
• Scrapes the corresponding website for additional content.
• Downloads token images (header and logo) and saves them locally.
• Saves token descriptions into text files.
• Saves Ethereum and Base contracts if they are "safe" to use (DYOR).
• Automatically runs every 10 minutes to update token information.

Installation

1. Clone the repository:

   git clone https://github.com/wafardev/dexscreener-scraper/
   cd dexscreener-scraper

2. Install the required packages:

   npm install

3. Save your API Keys for Etherscan and TTF Bot in a new .env file:

   ETHERSCAN_API_KEY=etherscan_api_key
   TTF_API_TOKEN=ttf_api_token

Usage

1. Ensure you have Node.js installed (version 14 or higher recommended).

2. Run the scraper:

   npm run dev

   For production use, consider using a process manager like PM2 (if installed):

   npm run prod

   Then run the logs with:

   npm run logs

   Restart the PM2 process with:

   npm run restart

   The PM2 process will run indefinitely in the background and won't halt if not deleted, so remove the process with:

   npm run delete

3. The scraper will run and check for new token profiles every 10 minutes, saving data into the downloaded-sites directory and saved contracts into the saved-contracts directory.

4. Cleanup any website containing errors and unsafe contracts using:

   npm run cleanup

Directory Structure

The downloaded data will be organized as follows:

downloaded-sites/
└── <token-website-hostname>/
    ├── index.html
    └── tokenInfo/
        ├── description.txt
        ├── header.png
        └── logo.png

saved-contracts/
└──  <chain-name>_<contract-address>_<safe-or-not-safe>.sol/

Functions

• getLatestTokenProfile(): Fetches the latest token profile data from Dexscreener API.
• scrape(url): Scrapes the specified URL and downloads the HTML content and resources.
• downloadHeaderAndLogo(outputDir, headerUrl, iconUrl): Downloads the header image and logo into the specified directory.
• saveDescription(outputDir, description): Saves the token description into a text file.
• fetchContract(tokenProfile): Fetches source code from Etherscan and runs the code through a static analysis tool for vulnerabilities.

Error Handling

The scraper has built-in error handling to ensure that it continues to run even if individual token profiles fail to process. Errors are logged to the console for debugging.

Contributing

Contributions are welcome! Please feel free to submit a pull request or open an issue for any suggestions or improvements.

Acknowledgments

• Dexscreener for providing the API.
• Node.js for enabling easy JavaScript server-side development.
• Puppeteer for web scraping capabilities.
• Etherscan for providing blockchain data.
• TTF Bot for their static analysis tool.