Page tree
Skip to end of metadata
Go to start of metadata

Purpose

We often introduce bugs when the layout of pages is unintentionally changed (MAGNOLIA-5029 -> MAGNOLIA-5132) or the dialogs were broken due to changes in javascript (MAGNOLIA-4829 -> MAGNOLIA-4895). Another problem is that the layout of pages between author preview and public are not the same (missing empty DIVs which anybody noticed - compare demo-project_author.png vs demo-project_public.png, empty optional areas rendered on public  MAGNOLIA-5227)  We should avoid this by adding tests for comparing sample screenshots from ce/ee bundles with current ones. This will also automate page rendering/layout QA.

What could be compared

  • webpages (public/author + with/without page editor)
    • samples module
    • demo-project
  • admincentral pages
  • admincentral dialogs

Concept

What we can use for these tests

 

  • Selenium

    Selenium is able to take screenshots already. All we need is to create pack of sample screenshots which will be compared with current ones.

    • + Selenium is able to take screenshot of whole page even if it's not completely visible in browser (scrollbar).
    • + We need to target ce/ee bundles to same location to avoid redundancy of sample screenshots
    • + We have tested such comparison already (see MAGNOLIA-5216)

 

  • External tool

    We could possibly use some external tool to do such comparison

    • + Independency on bundles
    • - Is anybody familiar with such tool already?
    • - Does any tool take screenshot of page behind the visible screen as Selenium does?

Implementation

 

We have selenium tests in bundles already.

  • 4.5 PageEditorTest uses Selenium1
  • 5 UI tests use Selenium2

UI tests use taking of screenshots already and we didn't meet any issues when testing of taking screenshots with Selenium1.

 

How would be screenshots saved?

 

  • current screenshots

    • Target folder: UI tests save screenshots into  magnolia-integration-tests/tests/target/surefire-reports/ folder. We should respect this pattern.
    • Screenshots names: UI tests name screenshots with this pattern: 
      ActivationUITest_activatePageResultInStatusChange_177_By.xpath%3A+%2F%2Fdiv%5Bcontains%28%40class%2C+%27item%27%29%5D%2F*%5B%40class+%3D+%27label%27+and+text%28%29+%3D+%27Pages%27%5D.png
      This is insufficient for out task and we need folder structure to remain traceability in such big tree as demo-project website:  magnolia-integration-tests/tests/target/surefire-reports/ComparingScreenShotsUITest/demo-project/about.png
      We would save save probably only screenshots which would fail on comparison, others would be deleted after each comparison sample x current.

  • sample screenshots

    • in every bundle:
      • git db would be too big,
      • update of samples would be not maintainable (4.5.x/5.0.x/5.x - ce/ee bundles).
    • in separate module - only integration tests would have dependency
    • locally on Hudson
    • in separate git repository

 

Comparison:

Binary comparison is not usable, we cannot expect 100% match. Better would be pixel by pixel comparison. We can compute % of wrong pixels this way and set threshold to suitable value (5% seems OK, 5% threshold passes for small pixel-differences but is able to find missing empty DIVs - MAGNOLIA-5227). We should convert the images to black&wait to avoid problems with color shades (empty gray placeholders in have different shades of gray, at least when testing on 4.5 author instance). This would also prevent problems with color changes. We can check layout in B&W without any side effects.

Target URLs

The lists of URL for comparison should be saved in /src/test/resources/targetURls folder as files:

  • website (website pages) as demo-project or Samples Module pages (/demo-project/members-area/registration/registration-confirmation)
  • dialogs (optionally with parameters i.e. .magnolia/dialogs/useredit?mgnlPath=/system/anonymous&mgnlRepository=users&mgnlRichE=true&mgnlRichEPaste=textarea&mgnlCK=1375962974071)
  • pages (admincentral pages i.e. .magnolia/pages/dmsZipUpload?repository=dms&path=/demo-project/img/logos/magnolia-logo&mgnlCK=1376047515166)
Browser dependency

Screenshots from different  browser/OS are different Because it's not possible to force developers to use the same browser/browser version we have to:

  • make separate sample screenshots packs for different browsers (unmaintainable)
  • run these tests only on Hudson

Set of tests

 

  • Minimum
    • compareWebsiteAuthorTest() - compare website pages from targetURLs/website list - author instance vs sample screenshots
    • compareWebsitePublicTest() - compare website pages from targetURLs/website list - public instance vs sample screenshots

  • Additionally
    • compareWebsiteAuthorVsPublic() - compare website pages from targetURLs/website list - author vs. public instance (no sample screenshots needed)

    • compareDialogsAuthorTest() - compare admincentral dialogs from targetURLs/website list - author instance vs sample screenshots
    • compareDialogsPublicTest() - compare admincentral dialogs from targetURLs/website list - public instance vs sample screenshots
    • compareAdmincentralAuthorTest() - compare admincentral pages from targetURLs/website list - author instance vs sample screenshots
    • compareAdmincentralPublicTest() - compare admincentral pages from targetURLs/website list - public instance vs sample screenshots

Adding new tests & Maintenance

If the layout would change intentionally and this is not so often (we can always change threshold if the test would be too sensitive)...test would fail, failed screenshots will be in surefire/ directory on Hudson, we can check them visually and if everything would be OK, we can upload them as samples.

To add new target URL for comparison:

  • upload new screenshot
  • update or add new list of URLs to check

 

  • No labels