I created Middle East GCC car database in 2016 to serve the numerous visitors from United Arab Emirates and Saudi Arabia that were coming on my website looking for a car database for their region. I paid my programmer partner to make a standalone scraping application (exe file) for DriveArabia.com and include 16 columns of technical specs. Took 1h 30min to scrap all. He made an universal scraping application in late 2016.
DriveArabia do not have individual pages for each engine, but just for model. Model pages are identical for all years you select, all engines offered during whole model production period (not just the year you selected) are displayed in table format. Some models display 0-100 kph and top speed as ranges, this because the model have multiple body styles with different specifications, and the website do not show body styles separate.
In summer 2017 DriveArabia had a major redesign, making standalone scraping application useless. They removed the field displaying all years a model was produced without changes, a must-have feature to distinguish where a generation ends and next one starts.
I used the universal scraper instead, configuring it to include all data from website, not just technical specs but also model overview, reviews, reliability, and image URL. Universal scraper had same speed but after scraping had to manipulate data for few hours to get right format. In 2018 my programmer improved universal scraper so the manual work is no longer required, data comes in the right format in just 1h 30min, allowing me to update database as often customers wants.
List of updates
An one-model sample published on 05 May 2016 and emailed 3 potential customers asking if they would purchase.
83 makes, 767 models, 5051 years (5 models with no years), as 13 May 2016 (experimental, not published).
83 makes, 771 models, 5190 years (5 models with no years), 9446 engines, 2605 engines if remove duplicate years, as 09 July 2016 (published for first time).
84 makes, 777 models, 5368 years (9 models with no years), 9775 engines, 2860 engines if remove duplicate years, as 17 August 2016 (first sale, first update).
85 makes, 783 models, 5390 years (6 models with no years), 9832 engines, 3032 engines if remove duplicate years, as 12 November 2016.
85 makes, 791 models, 5430 years (8 models with no years), as 02 January 2017.
86 makes, 803 models, 5448 years (5 models with no years), 9956 engines, as 07 April 2017.
89 makes, 843 models, 5537 years (14 models with no years), 10113 engines, 3198 engines if remove duplicate years, as 03 August 2017.
Due to changes in the source website, the previous scraper could not be used anymore so I used my partner’s universal scraper which did not produced database exactly in the right format so had to spend about 8 hours manually copy-pasting. Database expanded from 16 to 26 columns.
89 makes, 844 models, 5565 years, 10168 engines, as 26 October 2017, experiment on new website design.
89 makes, 845 models, 5571 years (13 models with no years), 10179 engines, as 10 November 2017.
90 makes, 895 models, 6050 years (10 models with no years), 10952 engines, as 20 April 2018.
90 makes, 908 models, 6099 years (9 models with no years), 11026 engines, as 05 September 2018. My programmer partner made a new scraper so manual copy-pasting is no longer required, allowing me to update more often.
90 makes, 908 models, 6110 years (9 models with no years), 11034 engines, as 07 October 2018, update done for someone who wanted other countries than UAE.
90 makes, 911 models, 6150 years (12 models with no years), 11128 engines, as 08 December 2018.
91 makes, 918 models, 6504 years (10 models with no years), 11752 engines, as 03 February 2019.
91 makes, 928 models, 6541 years (9 models with no years), 11766 engines, as 29 May 2019. I added “new cars only” packages with just last 3 years of cars (2018-2020), but due to this reason I had to raise prices of 1991-2020 packages.
91 makes, 935 models, 6623 years (11 models with no years), ? engines, as 1 October 2019.
92 makes, 938 models, 6684 years (11 models with no years), 12012 engines, as 9 October 2019.
93 makes, 947 models (1 make with no models), 6916 years (10 models with no years), 12411 engines, as 4 January 2020.
For few months DriveArabia was showing CloudFlare loading screen for 5 seconds then an error that website is not accessible… I waited hoping they solve the problem, until one day in August I activated VPN for other reason and incidentally checked DriveArabia and saw that it is working on foreign IP. CloudFlare problem was only for IPs of my country or whatever..
95 makes, 987 models, 7105 years (5 models with no years), 12772 engines, as 28 August 2020.
95 makes, 993 models, 7158 years (5 models with no years), 12819 engines, as 1 October 2020.
95 makes, 996 models, 7489 years (2 models with no years), 13391 engines, as 22 December 2020.
95 makes, 996 models, 7522 years (2 models with no years), 13431 engines, as 11 January 2021.
95 makes, 1007 models, 7558 years (3 models with no years), 13493 engines, as 17 May 2021.
95 makes, 1015 models, 7666 years (3 models with no years), 13659 engines, as 6 August 2021 (not published because I forgot to scrap twice to include also Arabic names)
95 makes, 1016 models, 7673 years (2 models with no years), 7654, 13670 engines, as 18 August 2021.
95 makes, 1040 models, 8075 years (2 models with no years), 14368 engines, as 25 January 2022.
96 makes, 1054 models, 8140 years (3 models with no years), 14462 engines, as 1 July 2022.
96 makes, 1057 models, 8144 years (4 models with no years), ? engines, as 27 September 2022.
99 makes, 1076 models, 8500>8472 years (2 models with no years), 15046 engines, as 13 January 2023.
Database update process takes about 90 min so it can be done anytime… if you think that too much time passed since last update, JUST ASK me to do an update TODAY.
