Content quality checks | Community
Skip to main content
aanchal-sikka
Community Advisor
aanchal-sikkaCommunity Advisor
Community Advisor
November 22, 2024
Solved

Content quality checks

  • November 22, 2024
  • 4 replies
  • 1149 views

The majority of our assets are ingested through RESTful APIs, and often the metadata fails to meet compliance standards. I am seeking recommendations for frameworks or approach that can be utilized to

  • validate metadata via RESTful services.
  • perform regular content quality checks for existing assets.
This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by aanchal-sikka

Thanks @tethich  @fanindra_surat @a_h_m_imrul @narendragandhi for your response. All the approaches seem valid for different use-cases.

Sharing my thought process, after borrowing your amazing suggestions.

 

Key Stages for Data Validation

  1. Existing Data:

    • Prioritize validation of severe issues that have a significant impact on metadata quality.
    • Address medium or low-impact issues progressively as part of a cleanup plan.

  2. Incoming Data:

    • Implement rules and workflows to ensure compliance before data ingestion.
    • Relax rules strategically when needed but maintain processes to identify and correct discrepancies later.

  3. Evolving Rules:

    • Introduce and enforce new validation rules as new business requirements, compliance needs, or metadata values emerge.

Leveraging Reports Over Time

Validation efforts should serve the needs of both technical and business teams:

  • Technical Teams:
    • Perform full-system health checks regularly to identify gaps across all assets.
  • Business Teams:
    • Validate and clean subsets of data in small or large batches.
    • Revalidate after cleanup to ensure compliance with updated rules.
    • Employ user-friendly tools to facilitate incremental cleanups without technical intervention.

Reports and Visualizations

  • Reports: Generate segmented reports tailored to specific teams, stakeholders, or metadata filters.
  • Visual Dashboards: Use tools like Power BI to create intuitive dashboards that visualize validation progress, compliance, and areas needing attention.

Technical Approaches for Data Validation

  1. Scheduled Health Reports: Automate periodic checks using schedulers to assess overall system health. 

  2. Extend ACS Commons Reports for validate assets + generate targeted reports by query, path, or metadata filters.

  3. External Validations before ingesting data: This would definitely have been a good approach to lighten AEM load. However, there are few challenges:

    • Dependency on other team to update the rules

    • We would still need an approach to identify issues in existing assets/ones updated manually/ gradual clean-up of medium-low priority data.


I am inclining towards Custom Reporting and Storage by using:

  1. "DAM Update Asset workflow + ACS Commons Reports" for medium/low compliance issues
  2. Rejecting assets with severe compliance issues.

Store validation results for non-compliant assets in /var nodes, enabling:

  • Extracting specific reports for Metadata-specific cleanup tasks.
  • Easy mapping to visual dashboards for better analysis

Please do share your thoughts if you see any challenges/improvements.

4 replies

Tethich
Community Advisor
TethichCommunity Advisor
Community Advisor
November 22, 2024

Hi @aanchal-sikka 

 

Is it possible to expand a little more the subject ? How does your current import process looks like ? How do all the parties involved in the ingestion integrate ? When and where do you expect the regular quality checks on the content to happen ? What does it mean for you that a metadata compliance is failing ? Some diagrams or screenshot also would not hurt.

    aanchal-sikka
    Community Advisor
    aanchal-sikkaCommunity AdvisorAuthor
    Community Advisor
    November 22, 2024

    @tethich 

     

    Thanks for the queries. Sharing details below:

    • The import is done by AEM by reading the Asset & related metadata from a location.
    • Integration is a pull mechanism from AEM, by reading S3 buckets.
    • Checks can be at Asset Import:
      • Assets to be rejected if severe compliance issue in metadata
      • Assets accepted with minor compliance issues in metadata. These should be reported by regular health checks.

    Non-compliance can be if metadata is from a specific set of values, format etc.

    Aanchal Sikka
      narendragandhi
      Community Advisor
      narendragandhiCommunity Advisor
      Community Advisor
      November 22, 2024

      Hi @aanchal-sikka 

       

      For the regular scheduled quality check, it can be a scheduled job which can get the list of updated assets in a time period. This will perform the specified checks and then add/update a field in metadata that keeps track of the last review date.

       

      For the API to fetch metadata of assets, you can explore this - https://developer.adobe.com/experience-cloud/experience-manager-apis/api/experimental/assets/author/

       

      Hope this helps!

       

      Narendra

       

        A_H_M_Imrul
        Community Advisor
        A_H_M_ImrulCommunity Advisor
        Community Advisor
        November 22, 2024

        Hello @aanchal-sikka ,

        I hope you're doing well.

         

        • Given the additional computational overhead, it might be better to perform validation or sanity checks outside of AEM, prior to asset ingestion, if feasible.

        • If the compliance violations occur when assets are exposed to traffic from the publisher, could we consider using an asset replication interceptor (such as a replication preprocessor) to validate and allow only compliant assets to be replicated?

        • To monitor faulty assets, we could set up a scheduled Sling job to generate reports that identify non-compliant entries.

        Let me know if this approach makes sense

          Fanindra_Surat
          Community Advisor
          Fanindra_SuratCommunity Advisor
          Community Advisor
          November 22, 2024

          Hi @aanchal-sikka -

           

          My thoughts -

          From what you shared below, you have a custom process setup in AEM that reads assets and metadata from a S3 location. Are you referring to the issues within this metadata that is stored separately or the ones like, xmp metadata that are extracted out of an asset? 

          If it is the separately managed metadata - Can you not validate or run your compliance check during the ingestion phase in your custom process?

          If it is the xmp metadata - you will need to create a custom process and configure it to be invoked as part of a DAM metadata writeback workflow itself or as a separate scheduler as per the need.

           

          Regards,

          Fani

            kautuk_sahni
            Community Manager
            kautuk_sahniCommunity Manager
            Community Manager
            November 25, 2024

            @aanchal-sikka Did you find the suggestions helpful? Please let us know if you require more information. Otherwise, please mark the answer as correct for posterity. If you've discovered a solution yourself, we would appreciate it if you could share it with the community. Thank you!

            Kautuk Sahni
            aanchal-sikka
            Community Advisor
            aanchal-sikkaCommunity AdvisorAuthorAccepted solution
            Community Advisor
            November 26, 2024

            Thanks @tethich  @fanindra_surat @a_h_m_imrul @narendragandhi for your response. All the approaches seem valid for different use-cases.

            Sharing my thought process, after borrowing your amazing suggestions.

             

            Key Stages for Data Validation

            1. Existing Data:

              • Prioritize validation of severe issues that have a significant impact on metadata quality.
              • Address medium or low-impact issues progressively as part of a cleanup plan.

            2. Incoming Data:

              • Implement rules and workflows to ensure compliance before data ingestion.
              • Relax rules strategically when needed but maintain processes to identify and correct discrepancies later.

            3. Evolving Rules:

              • Introduce and enforce new validation rules as new business requirements, compliance needs, or metadata values emerge.

            Leveraging Reports Over Time

            Validation efforts should serve the needs of both technical and business teams:

            • Technical Teams:
              • Perform full-system health checks regularly to identify gaps across all assets.
            • Business Teams:
              • Validate and clean subsets of data in small or large batches.
              • Revalidate after cleanup to ensure compliance with updated rules.
              • Employ user-friendly tools to facilitate incremental cleanups without technical intervention.

            Reports and Visualizations

            • Reports: Generate segmented reports tailored to specific teams, stakeholders, or metadata filters.
            • Visual Dashboards: Use tools like Power BI to create intuitive dashboards that visualize validation progress, compliance, and areas needing attention.

            Technical Approaches for Data Validation

            1. Scheduled Health Reports: Automate periodic checks using schedulers to assess overall system health. 

            2. Extend ACS Commons Reports for validate assets + generate targeted reports by query, path, or metadata filters.

            3. External Validations before ingesting data: This would definitely have been a good approach to lighten AEM load. However, there are few challenges:

              • Dependency on other team to update the rules

              • We would still need an approach to identify issues in existing assets/ones updated manually/ gradual clean-up of medium-low priority data.


            I am inclining towards Custom Reporting and Storage by using:

            1. "DAM Update Asset workflow + ACS Commons Reports" for medium/low compliance issues
            2. Rejecting assets with severe compliance issues.

            Store validation results for non-compliant assets in /var nodes, enabling:

            • Extracting specific reports for Metadata-specific cleanup tasks.
            • Easy mapping to visual dashboards for better analysis

            Please do share your thoughts if you see any challenges/improvements.

            Aanchal Sikka
              Terms & ConditionsAccessibility statement

              Loading footer...