Sorry for all Australian customers, I abandoned Australia Car Database to focus on increasing update frequency of European and American car databases that I can update myself and have higher sales volume. Reasons:

1. Australia Car Database was by far, the most difficult database I ever made, after few customers requested me to scrap data from Redbook, a website with anti-scraping technology so I had to pay third-party programmers (some being not trusted, gone away leaving me without updates), and given by small population of Australia, my efforts never paid off from sales, then in January 2020 Redbook added CAPTCHA that made data scraping impossible.

2. In July 2020 been contacted by a Redbook lawyer asking me to cease & desist selling Australia. Redbook TOS forbid scraping data:

Personal and Non Commercial Use Only

(a) Use of the RedBook Website is for your personal and non-commercial use only. Except for the Material held in your computers cache or a single permanent copy of the Material for your personal use you must not without our prior written approval:
– use any automated process of any sort to query, access, retrieve, scrape, data-mine or copy any Material on the RedBook Website or generate or compile any document, index or database based on the Material published on the RedBook Website;

If you want to use Redbook data, you must contact Redbook for a quote (will cost you 4 digits of $ depending by your company size and amount of data required). Many startup companies unable to afford Redbook price tried to pay few hundreds $ to me or other freelancers to code a scraper to get data in Excel format. I did not wanted to steal customers from Redbook, I wanted to help startup companies who were anyway not able to afford buying from Redbook (when they grow they could become Redbook direct customers and get realtime updates). However this turned too risky, Redbook can make legal actions against anyone scraping data illegally.

If you want an Australia car database from me, we need find a website that I can legally scrap data from (other than Redbook, Carsales, Carsguide – all use same Redbook API thus cannot be scraped for both technical and legal reasons). Any suggestions?

History

I wrote this page in 2016 as possible future project, inviting people to contact me if are interested in a car database for Australia and suggest websites to scrap data from, other than Redbook because its TOS forbid scraping. I informed visitors that Redbook is selling data so would be ILLEGAL for me to scrap data from a seller and sell on my behalf, and asked them to suggest me other websites to scrap data from, but once few visitors insisted me to scrap from Redbook because is the BEST, I had no other choice to help them…

I created Australia car database in June 2017 using Octoparse.com, a FREE scraper slow and buggy, had to run in small batches of 1000-2000 cars and took 2 weeks to complete project. I updated every 3-4 months, scraping only last 2 years of models, each update taking 2-3 days.

In February 2019 I worked with an Australian student who made a Python script that took only 2 days to scrap all cars, gave me a .py scraper with errors, which I reported and he fixed afterwards but instead of giving me final scraper, he setup scraper to run on his server (monthly schedule) and gave me login info to download CSV. Everything looked perfect, but in April 2019 he gone away, never replied emails, never signed in Skype, and server was closed, leaving me and my customers with no updates.

In July 2019 I hired a Python programmer from India who turned to be a scam artist, lied me that has made over 80 databases using Python, which turned to be free/open data downloaded from internet. He spent 4 months “working” and kept asking me payments, but never gave me an error-free script. I started scraping with his script in November 2019, it took 20 days to finish so only in December 2019 I was able to deliver to customers an update that still had errors.

In January 2020 Redbook added a CAPTCHA that appear every time I access website using a new browser, blocking my scraper. The indian, instead of apologizing for his errors and try to do better job next time, refuse to help me because I blamed his inexperience for errors he made in 2019. I tried to ask help from 4 programmers to add a cookie feature in his Python script to bypass captcha but nobody was able to succeed. One sent me this reply:

Sat, May 30, 8:02 PM
We looked at this in detail, and discoved that redbook are using a sophisticated AI based protection system called datadome, its not just the cookie that it is using to protect the data, its a load of other meterics, like access frequency, coverage and other technical metrics.
You can read about it here: https://datadome.co
We decided after some experiments that it would not viable to bypass this system.

21 thoughts on “Australia”

  1. Hello

    I’m looking for Australian car specification data from 1970 to current year

    Database should contain following fields

    Make, Model,Year, Body Type, Transmission and Variant

    Thanks

    1. 2 comments in same day…. S and J (full names removed at request), are you from same team?

      A week ago another 2 people contacted me via LIVE CHAT about Australian car database, so I decided to start experimenting various scraping methods on various sites.
      C-TEST.xls (link removed)
      R-TEST.xls (link removed)
      I can sell you specific fields only.

      carsales.com.au have used cars classifieds only and no specs for every car sorted by make and model. So it’s useless

      1. 2 comments in same day…. S and John Romaine, are you from same team?

        No mate.

        Coincidence lol

        Let me have a look at your XLS files.

    2. Hi .
      Hi.
      I would interested in the Australian DB but I am just doing something for fun so no commercial intentions. I could pay something but not much. Carsales does have specs in another tab. Not sure about the alpha order. I would be also interested in the global one. But I just want basic info. What price ? 40 euros ?
      Thanks,
      Frank.

      1. I can’t believe how many comments from Australia I get in a short period of time, this page being here since May 2016 so for 1 year I did not got a single comment!

        I noticed that CarSales.com.au and CarPoint.com.au have a Research section, they not just show same data like Redbook but are also same pain in the ass at scraping, my partner scraper takes 50 seconds per page and jumps some pages (which at other websites takes max 2 seconds per page and up to 5 pages per second). So I use a third-party scraper which takes about 10-15 seconds per page = takes over 300 hours to scrap, but I run 4 scraps at same time thus reduce time to 7 days.

        For global database with basic info only you can buy https://www.teoalida.com/cardatabase/car-models-list/

        1. Hi.

          Its a coincidence that I just happened to have some spare time and wanted to do a little project. I got laid off from working at Oracle for 26 years. While I know a bit about DBs I dont know much about scraping.

          Well I assume the time effects the cost. I just want basic data. Any odeas on the prices I asked you for ?
          .btw where are you located ?

          Cheers,
          Frank.

      2. Than you to all people who showed interest in this project encouraging me to start building Australian car database, 2 of them contacted me via LIVE CHAT on 26 May, then S and John Romaine who left comments on 3 and 4 June.

        Database finalized and published today!

        Feel free to contact me and negotiate price if you want a specific set of columns and historical coverage!

      3. After months of trying to gather a Australian Car database I found teoalida’s website. At the time he didn’t have a Australian database but was ready for the challenge.
        After downloading an checking the database today, it is extremely detailed, complete and im extremely happy with the purchase. If anyone else is looking for this database, it wont disappoint.

        Thanks again for your help mate.

      4. Hi

        I am looking for an Australian vehicle database and came across your website. I have a looked at your sample database. It has more information than I need, but at the same time it is missing some critical information. I need a database that will have the following information:
        – overall length
        – overall width (no mirrors)
        – overall height
        – wheelbase
        – front overhang
        – rear overhang
        – turning radius/diameter – wall to wall
        – turning radius/diameter – kerb to kerb
        – ground clearance (unladen)
        – ground clearance (fully laden)
        – front or rear axle lock to lock time
        – front truck width
        – rear track width
        – front tire width and diameter
        – rear tire width and diameter
        – approach angle (front underside of vehicle)
        – departure angle (rear underside of vehicle)
        – dimensions of side mirrors (including any attachments so it can fixed to the body of the vehicle).

        I realise the above list may be quite extensive. It is not necessary to have all the information, but it would be good to have. Order of importance is top to bottom. To give you a better idea our need for the database, we work as consulting traffic engineers and we design car parks. From time to time, we have to assessed whether a car park will be practicable or not. We need to run vehicle path simulation using vehicle dimensions. The software we used requires the above as input.

        Can you give me a quote to create such a database?

        I assume the data you collected relates to vehicles available for sale in Australia, not necessary registered vehicles? Are you able to create database for registered vehicles.

        1. I already included all possible data that could be scraped from the source website. If you need other data you should suggest another website to scrap data from. If you want a custom package with just few columns I can sell you at lower price than full specs.

            1. Probably 0.3 cents per car, select exact columns you want. Use live chat in lower-left corner of screen so we can communicate faster.

          1. Hi there, I need a custom database. Can you please provide a quote for the Australian car database between 1960-2018 with the following:

            Full car name, make, model, year, body description.

            Thanks,

            Ryan

          2. Hello I’m looking for a lw16r tractor it was stopped being made around the 1960s and we have one our selves left on a property we bought. I would like to get it going again and I need to know more about it because when I google it it comes up with the first model lw12. The company who makes them is LeTourneau’s Westinghouse and was manufactured in Rydalmere in New South Wales, Australia. can you please find out a website or any information you would have on them thanks George

            1. Do you have any problem with chat box? You left the chat after short time.

              Chat started on Saturday, December 22, 2018 11:49:11 AM

              (11:49:11 AM) *** Visitor 88341860 has joined the chat ***
              (11:49:13 AM) Teoalida (site owner): Hi there… are you looking for a specific database? Let me know if I can help you!
              (11:50:25 AM) Visitor 88341860: Hello I’m trying to find information about the lw16r tractor but it was made in New Zealand wondering if you have information on it
              (11:50:52 AM) *** Teoalida has joined the chat ***
              (11:51:10 AM) Teoalida: I do not have any database with tractors right now, but I can create one scraping data from https://www.ritchiespecs.com/
              (11:51:18 AM) *** Visitor 88341860 has left ***

          3. Hi,
            We are currently looking for a quality Auto Data base supplier, just wondering how up to date are you data files, with respect of course,

            Australian needed to start with,

            Other countries will be required after the initial order & purchase.

            Looking forward to your reply.

            Kind Regards
            Anthony

Leave a Reply

Your email address will not be published. Required fields are marked *