Perplexity Pro's AI absolutely aced my coding tests

A number of weeks in the past, I ran my customary suite of programming checks towards the free model of the Perplexity.ai chatbot. On the finish of the article, I provided to run checks towards the $20/mo Professional model if sufficient of you had been . I did get some requests, so that is what we’re doing right here.

Like most different professional variations, to make use of Perplexity Professional, it’s important to create an account. You may sign up utilizing both Google or Apple auth strategies or a SAML sign-in. Alternatively, you may create an account utilizing your electronic mail handle, which is what I did.

Sadly, the location does not appear to provide you any approach to set a password or any type of multifactor authentication. You are despatched an electronic mail with a code, and that is it. I do not thoughts getting an electronic mail code, however I am actually disturbed by net apps relying solely on an electronic mail code with out, at the least, a password. However that is what Perplexity.AI is doing.

- Advertisement -

The opposite fascinating facet of Perplexity Professional is its cornucopia of AI fashions. As you may see within the picture under, you may select between a variety of totally different fashions, based mostly on the type of work you have got. I selected Default to see what that did with the checks. After operating the checks, I requested Perplexity Professional what mannequin it used for them, and it informed me ChatGPT GPT-4.

And with that, let’s run some checks.

1. Writing a WordPress plugin

This problem is a reasonably easy programming activity for anybody with a modicum of net programming expertise. It presents a person interface within the administration dashboard with two fields: one is a listing of names to be randomized, and the opposite is the output.

The one actual gotcha is that the listing of names can have duplicates, and quite than eradicating the additional names, its directions are to verify the duplicate names are separated from one another.

- Advertisement -

This was an actual, requested operate that my spouse wanted to make use of for her e-commerce web site. Each month, they do a wheel spin and a few folks qualify for a number of entries.

2. Rewriting a string operate

For every take a look at, I open a brand new session with the AI. On this take a look at, I am asking the AI to rewrite a block of code that had a bug. The code was designed to validate the enter of {dollars} and cents, which ought to comprise a sure variety of digits earlier than the decimal level, a doable decimal level, and two digits after the decimal level.

Sadly, the code I shipped solely allowed integer numbers. After a few person experiences, I made a decision to feed the code to the AI for a rewrite. My code makes use of common expressions, that are a formulaic manner of specifying a format. Common expressions themselves are enjoyable, however debugging them isn’t.

Within the case of this take a look at, Perplexity Professional did a great job. The resultant validation code correctly flagged objects that didn’t match the format for {dollars} and cents, permitting as much as two digits after the decimal.

Listed here are the mixture outcomes of this and former checks:

- Advertisement -

Perplexity Professional: Succeeded
Perplexity: Succeeded
Claude 3.5 Sonnet: Failed
ChatGPT utilizing GPT-4o: Succeeded
Microsoft Copilot: Failed
Meta AI: Failed
Meta Code Llama: Succeeded
Google Gemini Superior: Failed
ChatGPT utilizing GPT-4: Succeeded
ChatGPT utilizing GPT-3.5: Succeeded

3. Discovering an annoying bug

This take a look at had me stumped for just a few hours. Earlier than it was a take a look at, it was a bug within the code for an precise product. The issue was that no matter was going improper wasn’t associated to any apparent logic or language subject.

Being severely pissed off, I made a decision to feed ChatGPT the code in addition to the error dump and ask it for assist. Fortuitously, it discovered what I had executed improper and gave me steering on what to repair.

The explanation I am together with this within the set of checks is as a result of the bug wasn’t in language or logic, it was in information of the WordPress framework. Whereas WordPress is in style, framework information is usually thought of the folklore of a programming surroundings, one thing handed down from developer to developer, quite than one thing that may be rigorously realized by a information base.

Nevertheless, ChatGPT, in addition to Perplexity and now Perplexity Professional, did discover the issue. The error was as a consequence of a parameter calling subject buried within the framework itself. The apparent reply, which you may give you strictly by studying the error messages generated by the code, was truly improper.

To resolve it, the AI needed to present a deeper understanding of how all of the methods work collectively, one thing with Perplexity Professional did efficiently.

Listed here are the mixture outcomes of this and former checks:

Perplexity: Succeeded
Perplexity Professional: Succeeded
Claude 3.5 Sonnet: Succeeded
ChatGPT utilizing GPT-4o: Succeeded
Microsoft Copilot: Failed
Meta AI: Succeeded
Meta Code Llama: Failed
Google Gemini Superior: Failed
ChatGPT utilizing GPT-4: Succeeded
ChatGPT utilizing GPT-3.5: Succeeded

4. Writing a script

Properly, that is fascinating. Perplexity Professional handed this take a look at, however the free model of Perplexity failed after I examined it a few weeks in the past. So, yay!

However let’s dive into this a bit. The problem right here is that I ask the AI to jot down a script that intersects three environments: the Chrome DOM (doc object mannequin), AppleScript (Apple’s native scripting language), and Keyboard Maestro (a really cool Mac automation software that is pretty obscure, however to me, mission-critical).

A lot of the AIs failed as a result of they did not have any data on Keyboard Maestro of their information bases and, as such, did not give the mandatory code for the script to do what I wished.

Solely Gemini Superior and ChatGPT utilizing GPT-4 and GPT-4o handed this take a look at till now. In answering the query, Perplexity Professional supplied a Professional Search view. As you may see, the Professional Search view did a seek for “Keyboard Maestro AppleScript Google Chrome tabs.” It additionally used the principle Keyboard Maestro discussion board as a supply, which is the very best supply for getting Keyboard Maestro coding assist.

The outcome was a hit.

Listed here are the mixture outcomes of this and former checks:

Perplexity Professional: Succeeded
Perplexity: Failed
Claude 3.5 Sonnet: Failed
ChatGPT utilizing GPT-4o: Succeeded however with reservations
Microsoft Copilot: Failed
Meta AI: Failed
Meta Code Llama: Failed
Google Gemini Superior: Succeeded
ChatGPT utilizing GPT-4: Succeeded
ChatGPT utilizing GPT-3.5: Failed

General outcomes

Listed here are the general outcomes of the 4 checks:

As you may see, Perplexity Professional joins solely ChatGPT with GPT-4 and GPT-4o as having an ideal rating of 4 out of 4 succeeded. After operating my checks, I checked with Perplexity Professional’s AI and it knowledgeable me it used GPT-4 to investigate and reply to my checks.

On condition that GPT-4/4o is the one AI that nailed all 4 of my checks earlier than, this is sensible. Thus far, I have never discovered some other mannequin that may totally and appropriately cross all 4 programming checks.

In case you select Perplexity Professional, I can pretty confidently state that it ought to be capable of do a great job of serving to you program.

Have you ever tried coding with Perplexity, Copilot, Meta AI, Gemini, or ChatGPT? What has your expertise been? Tell us within the feedback under.

You may comply with my day-to-day venture updates on social media. Make sure to subscribe to my weekly replace e-newsletter, and comply with me on Twitter/X at @DavidGewirtz, on Fb at Fb.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, and on YouTube at YouTube.com/DavidGewirtzTV.