I put the free version of Perplexity.ai through my coding tests – here’s what happened

Published on:

I’ve examined the coding capabilities of many generative AI instruments for ZDNET — and this time, it is the flip of Perplexity.ai.

Perplexity looks like a cross between a search engine and an AI chatbot. Once I requested Perplexity the way it differs from different generative AI instruments, the bot mentioned it makes use of real-time info entry, indexing the net each day. Customers can slim searches by asking Perplexity to deal with sources or platforms.

The free model of Perplexity is pretty restricted. It makes use of OpenAI’s GPT-3.5 mannequin for evaluation, solely permits 5 questions a day, and whereas it helps doc uploads, these uploads are restricted to 3 per day.

- Advertisement -

The Professional model of Perplexity is $20/month. That model permits for limitless “fast” searches, 600 Professional searches per day, and the selection of AI mannequin. You possibly can select from GPT-4o, Claude 3, Sonar Giant (LLaMa 3), and others. The Professional model additionally gives $5/month in API credit.

We determined to forego the Professional and run the free model for our preliminary take a look at of Perplexity’s programming prowess. I’ve run these coding assessments towards AIs with diversified outcomes. If you wish to observe alongside, level your browser to ‘How I take a look at an AI chatbot’s coding capability – and you may too’, which incorporates all the usual assessments I apply, explanations of how they work, and particulars on what to search for within the outcomes.

Now let’s dig into the outcomes of every take a look at and see how they examine to earlier assessments utilizing Claude 3.5 Sonnet, Microsoft Copilot, Meta AI, Meta Code Llama, Google Gemini Superior, and ChatGPT.

1. Writing a WordPress plugin

This problem asks a number of issues. First, it asks the AI to create a consumer interface for getting into strains to be randomized (however not de-duped). Then the take a look at requires the AI to create a button that not solely randomizes the record however makes positive any duplicate gadgets are offered such that they don’t seem to be subsequent to one another within the ensuing record.

- Advertisement -
See also  Databricks expands Mosaic AI support for generative AI apps

To this point, most AIs, aside from Meta Code Llama, have created a reasonably cheap UI. Some have been extra enticing than others, however all of them did the job.

Nevertheless, solely ChatGPT (3.5, 4, and 4o) produced the right randomized output. A lot of the different AIs simply offered a button which, when clicked, did nothing.

Perplexity labored. It produced a UI correct to the spec and the Randomize button labored and separated duplicate strains.

Listed here are the combination outcomes of this and former assessments:

  • Perplexity: Interface: good, performance: good
  • Claude 3.5 Sonnet: Interface: good, performance: fail
  • ChatGPT GPT-4o: Interface: good, performance: good
  • Microsoft Copilot: Interface: sufficient, performance: fail
  • Meta AI: Interface: sufficient, performance: fail
  • Meta Code Llama: Full failure
  • Google Gemini Superior: Interface: good, performance: fail
  • ChatGPT 4: Interface: good, performance: good
  • ChatGPT 3.5: Interface: good, performance: good

2. Rewriting a string operate

This take a look at fixes a validation operate that checks for {dollars} and cents. 

My unique code was in error, permitting solely integer {dollars}, however no cents. I came upon when a consumer submitted a bug report. I initially fed the inaccurate code to ChatGPT, which did a superb job of rewriting the operate to permit greenback quantities and two digits to the correct of the decimal level.

Perplexity additionally handed this take a look at.

- Advertisement -

The code it generated may have been tighter, but it surely labored. In a case the place the string offered by the consumer contained solely zeros, Perplexity’s implementation eliminated all the pieces. To compensate, Perplexity checked for zero first. 

This strategy is workable, however the common expression Perplexity generated may have been written to account for this variation. It is a easy implementation selection and lots of certified programmers would have taken both path, so Perplexity’s strategy is suitable.

See also  Google adds ‘Web’ search filter for showing old-school text links as AI rolls out

Perplexity’s code accurately examined the submitted information to make sure it matched the {dollars} and cents format. The code then transformed the string to a quantity. It additionally checked if the quantity parsed was legitimate and non-negative.

General, Perplexity produced strong code. Listed here are the combination outcomes of this and former assessments:

  • Perplexity: Succeeded
  • Claude 3.5 Sonnet: Failed
  • ChatGPT GPT-4o: Succeeded
  • Microsoft Copilot: Failed
  • Meta AI: Failed
  • Meta Code Llama: Succeeded
  • Google Gemini Superior: Failed
  • ChatGPT 4: Succeeded
  • ChatGPT 3.5: Succeeded

3. Discovering an annoying bug

A bug in my code confused me, so I turned to ChatGPT for assist. Because it turned out, the supply of the issue was not intuitively apparent, which is why I missed it.

A parameter passing error requires data of how the WordPress framework capabilities. I missed the bug as a result of PHP appeared to suggest the issue was in a single a part of the code when, in reality, the difficulty was how the code transitioned by a WordPress-specific operation.

Perplexity discovered the issue and accurately identified the repair.

Listed here are the combination outcomes of this and former assessments:

  • Perplexity: Succeeded
  • Claude 3.5 Sonnet: Succeeded
  • ChatGPT GPT-4o: Succeeded
  • Microsoft Copilot: Failed
  • Meta AI: Succeeded
  • Meta Code Llama: Failed
  • Google Gemini Superior: Failed
  • ChatGPT 4: Succeeded
  • ChatGPT 3.5: Succeeded

4. Writing a script

This closing take a look at analyzes the extensiveness of the AI’s data base. The take a look at asks for code to be generated that requires data of the Chrome doc object mannequin, AppleScript, and a third-party Mac scripting software known as Keyboard Maestro.

Perplexity didn’t seem to find out about Keyboard Maestro, so it didn’t write the required name to the scripting language to retrieve the worth of a variable.

See also  IBM CEO praises real open source for enterprise gen AI, new efforts emerge at Think 2024

Perplexity additionally made the identical mistake Claude 3.5 Sonnet made, producing a line of AppleScript code that will end in a syntax error message upon operating. This error indicated a lack of know-how about how AppleScript ignores higher and decrease case, and the place it considers the case of a string when evaluating two values.

Listed here are the combination outcomes of this and former assessments:

  • Perplexity: Failed
  • Claude 3.5 Sonnet: Failed
  • ChatGPT GPT-4o: Succeeded however with reservations
  • Microsoft Copilot: Failed
  • Meta AI: Failed
  • Meta Code Llama: Failed
  • Google Gemini Superior: Succeeded
  • ChatGPT 4: Succeeded
  • ChatGPT 3.5: Failed

General outcomes

Listed here are the general outcomes of the 4 assessments:

General, Perplexity did nicely. I assumed the AI would possibly fail the fourth take a look at, as a result of ChatGPT 3.5 did, and the free model of Perplexity makes use of the GPT-3.5 mannequin.

I used to be stunned by these outcomes as a result of Microsoft’s Copilot can be supposed to make use of OpenAI’s AI engine, however Copilot failed at just about all the pieces. Perplexity mirrored the GPT-3.5 outcomes, which is smart for the reason that free model makes use of GPT-3.5.

Let me know if you wish to see how Perplexity Professional performs. If I get sufficient requests, I will join But One other Month-to-month AI Price and run some assessments.

Have you ever tried Perplexity’s free model or its Professional model? Tell us within the feedback beneath.


You possibly can observe my day-to-day challenge updates on social media. Make sure to subscribe to my weekly replace publication, and observe me on Twitter/X at @DavidGewirtz, on Fb at Fb.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, and on YouTube at YouTube.com/DavidGewirtzTV.

- Advertisment -

Related

- Advertisment -

Leave a Reply

Please enter your comment!
Please enter your name here