Expand my Community achievements bar.

Querying service: urgently needs dramatic improvement in querying speed

Avatar

yuhuisg
Community Advisor

8/22/22

Description - A simple query takes an unexpectedly long time to complete. See examples below. This indicates that AEP Queries is just not ready for widespread, "production" use, really.

Why is this feature important to you - 99% of writing a correct, working query is figuring out the way to write it. This requires a lot of iteration from a simple "SELECT * FROM table" to the final query. Querying speed is especially important here, so as to help the analyst improve upon the query until the final one is obtained. Any time spent waiting for a query to complete is time that could have been used to more productive use. Without this speed increase, Adobe can expect all of its AEP customers to flock to Google BigQuery and never look back.

How would you like the feature to work - Improve the querying engine's processing from minutes to milliseconds.

Current Behaviour - See examples.

SELECT * FROM table LIMIT 10;  --> 2.5 minutes

image.png

SELECT * FROM table WHERE table.eventType = "something" ORDER BY timestamp LIMIT 10; --> 10 minutes --> and to add insult to injury, the query failed.

image (1).png

SELECT * FROM table WHERE table.eventType LIKE "something" LIMIT 10; --> and yup, only to find out that it failed again.

yuhuisg_1-1661161203773.png

1 Comment

Avatar

Danny-Miller
Employee Advisor

8/22/22

@yuhuisg I hear you.

A few things that sometimes help:

  1. Add a Batch_ID to WHERE
  2. Replace
    1. SELECT * with
    2. SELECT [field1], [field2], [field3], [field4]
  3. Keep your LIMIT
  4. If you know you have a long running query, use CTAS 

Hope that helps a little.

Reply