Learning web scraping in late 2015 allowed me to quickly create new databases with less effort than if I were to enter data manually as I did before for European car database. Using scraping technology I created additional car models databases for America, India, Middle East, Australia, as well as other databases (world countries, mobile phones, etc).
However, I will NOT entertain any requests of creating car parts database, because assuming 10,000 car make, models and engines combinations, each one having few thousands parts, the required database would be an overkill and NOT profitable. See below examples:
Tuning database (9,000 rows and 20 columns) took 2 hours to create (less than 1 second/page) and sold to multiple people for $100 ($50/hour of scraping).
German car database (100,000+ rows and 200+ columns) took 100 hours to create (3 seconds/page) and sold to multiple people for $500 ($5/hour of scraping).
If a website have 36,000,000 car parts, 10,000 hours = 417 days = 14 months would be required to complete database (assuming 1 second/page), I would need to sell for $50,000 to respect minimum $5/hour, a price that nobody will afford.
Later research indicated that number of car parts may exceed 100 millions.
Several customers advised me to make car parts database just for particular makes, models or parts. Doing this way will NOT make job profitable, but contrary, would be difficult to sell same database to multiple people, since each one requiring different cars or parts.
In conclusion: I should forget idea of making car parts database and stick on many small databases (1,000 to 100,000 rows) selling at prices 50 to 500 euro (the situation may change if I get a bunch of servers for scraping instead of scraping from my computer).
History
In 2016 I was thinking about a possible next step, I started studying several car parts websites and talked with various potential customers, deciding together that www.onlinecarparts.co.uk is one of the best, but number of car parts seems too big to make this project viable, also I saw this warning:
The data shown here, especially the complete database, may not be copied. It is strictly prohibited to duplicate the data and database and distribute the same, and/or instruct third parties to engage in such activities, without prior consent from TecAlliance. Any use of content in a manner not expressly authorized constitutes copyright infringement and violators will be prosecuted.
I also attempted to scrap Make > Model > Year > Engine breakdown from rockauto.com but I got IP blocked after 10 minutes.
Think yourself, car parts data providers are million-dollar businesses with many employees, they can and have reasons to invest in anti-scraping measures, so the only way to obtain data is to buy from original author rather than hiring freelancers (and put me in copyright troubles of scraping from websites that prohibit copying data).
If you decide to buy from TecAlliance.net, please inform them that you were referred by Teoalida. One customer informed me that they charge $10,000/year. If you know other car parts data providers, please inform me, I want to contact them and propose collaboration, to advertise their services on my website and get paid a commission. TecAlliance rejected my proposal, sending me following reply:
Good afternoon
Many thanks for your inquiry. I am responsible for central European market and I received your email form my colleagues from TecAlliance HQ. As far I know, We bought some data from you, and until now it was the only business we made.
Frankly speaking we do not advertise and resell in way you describe, because our products are very specific. TecDoc Catalogue data is not a mysql database or a file in xls format.
Our customer have to sign contract always directly with us and they receive data form us directly, unless they work with our resellers, who integrated our data into their solution.
I think that is a misunderstanding. I did not meant to take TecDoc Catalogue and sell on my behalf, I wanted to put a banner advertisement with referral code so my website visitors can buy directly from TecAlliance and get a referral commission like how I do with https://www.teoalida.com/cardatabase/wheels-and-tires-size/
A desperate customer (it’s FUNNY how he insist to pay me for a service I DON’T offer)
Chat started on Sunday, October 04, 2020 3:57:07 PM
(3:57:07 PM) *** Visitor has joined the chat ***
(3:57:09 PM) Teoalida (site owner): Hello visitor from Qatar, are you looking for a specific database? Let me know if I can help you!
(4:05:12 PM) Visitor: Hello
(4:05:27 PM) *** Teoalida has joined the chat ***
(4:05:29 PM) Teoalida: what’s up
(4:06:07 PM) Visitor: Looking for spare part database to use it in my upcoming e-commerce store for spare parts
(4:06:31 PM) Visitor: Any database u offer for spare part items u can offer?
(4:06:47 PM) Teoalida: I don’t have car parts database, but I have car parts shops who bought databases from me to make drop-down boxes make > model > year > engine… for example https://www.teoalida.com/cardatabase/car-models-engines/
How many parts are in a car? Multiplied with number of car models, would require dozens million rows, imagine how much effort would take to make such large database and will be not profitable for me. As curiosity, what is your budget?
(4:07:24 PM) Visitor: How much you want? 😉
(4:07:52 PM) Visitor: I will require model starting from 2004 – till now
(4:08:47 PM) Visitor: Will give you list of car models required
(4:09:12 PM) Visitor: So not all models are required
(4:12:23 PM) Visitor: U here?
(4:13:56 PM) Teoalida: I don’t understand how I can help you
(4:14:08 PM) Teoalida: unless you want to buy databases with car specs that I done already
(4:14:51 PM) Visitor: Can you make that database for me?
(4:14:55 PM) Visitor: I would pay
(4:18:15 PM) Teoalida: read my above message again, I am NOT going to make car parts database for anyone
(4:18:40 PM) Teoalida: even if you want just a couple of models, someone else will not like a incomplete database, or he will want another couple of models so I need to work again
(4:19:12 PM) Teoalida: read more www.teoalida.com/cardatabase/car-parts/
(4:20:40 PM) Visitor: HOW MUCH YOU WANT FOR IT???
(4:20:51 PM) Visitor: I will pay
(4:22:11 PM) Visitor: Just tell me how much you want to do it for me ?
(4:23:16 PM) Teoalida: paying me for a new project won’t make my life better, I am already tired of staying ~12 hours/day at computer
(4:23:52 PM) Visitor: Ok i want to hire you 😁
(4:24:10 PM) Teoalida: too many people want to hire me https://www.teoalida.com/about/aboutme/#hiring
(4:25:59 PM) Visitor: Ok , can you connect me with your partners pho can do it ?
(4:26:30 PM) Visitor: “I won’t spend my personal time helping you, instead I connect you with my partners offering services starting at $10/hour (useful if you need long-term workforce)”
(4:27:22 PM) Teoalida: hmm, I don’t think that they are willing to take a project of THIS SIZE, anyway where do you want to source data from and how many rows do you expect to be in total?
(4:29:55 PM) Visitor: I have no idea from where this giant website got their data and their part vechial fittment. I am tired of searching and decided to hire some and pay
(4:32:28 PM) Visitor: I dont dont to have ALL parts details ,i want the data of less 70 parts from each car which are the major parts
(4:38:13 PM) *** Visitor has left ***
Well advised
What if create a separate database with all the car parts for every particular model? start with the most in demand.
Doing model by model will take even more effort to COMPLETE than doing all at once, and many customers wouldn’t like to buy an incomplete database.
There are almost 8 billion people on Earth, over 2000 visit my website every day, ~20 contact me every day, some just saying thanks for info, some requesting me to do various projects. Do I am the only one who should undertake such megalithic project?