MarketInsight Product Scraping Platform

Django Based Web Application For Automating Scraping Data

• Multi browser support (Chrome Native)
• Quick access to user scrap requests and scraped data.
• Two step authentication for better privacy.
• Manipulation over Scraped Data.
• Stylized User Interface for better experience.
• Threading for handling multiple scrap processes.

V2 Release

• Scraping stabilised
• Thread removed and Celery implemented.
• Required files for docker has been added to the project.

Upcoming Releases

• Local marketplace's integration (Ebay, Hepsiburada)
• Major upgrade in UI.
• Better visibility over scraped data.
• Safari support for Apple users.
• Token system for premium scraps.
• Premium scraps that allows users to scrap the data the way they want and reshape on flight.
• Excel export for scraped data.
• Different scraping methods for getting different kind of data for user needs.

How to Run the Project

1. Create and activate a virtual environment (recommended):

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

2. Install dependencies:

   pip install -r requirements.txt

3. Apply migrations:

   python manage.py migrate

4. Create a superuser (optional, for admin access):

   python manage.py createsuperuser

5. Run the development server:

   python manage.py runserver

6. Run the worker :

   celery -A product_crawler worker --loglevel=info -P solo
   

   (Not required if you build with docker)

How to Test

To run the test suite and ensure everything is working as expected:

python manage.py test

You can also run tests for a specific app:

python manage.py test <app_name>

Replace <app_name> with accounts, main_page, or product_amazon_crawler as needed.

Main Page

• Main page allows users to navigate through platform easily, here they can go to check their scrap requests, their status, and scraped data's detail.

Scrap Requests

• Page allows users to create new scrap request on selected platform
• User also can check its ongoing scraps and their statuses on listing.

Scraped Data

• Page allows users to see scraped data and its details.
• User also can check its details clicking the details button on the list.