About Project
A Scrapy crawler that scrapes millions of courses available on websites like Udemy, Coursera, khan academy, etc. Not just courses, it also scrapes course details like a respective tutor, course price, likes, dislikes, comments, etc. The script takes care of IP routing and VPN. Data scraped is stored in a MySQL database. Currently, a total of 13 large websites are being scraped, which is used by scientists for data analysis.
Backend workflow
● Scrape data from various educational websites
● Store data into MYSQL database
● Render the same for data analysis purpose
Technology used
● Python, Scrapy, Django, MySQL
Our Accomplishments
Our Client
