Purpose
We often introduce bugs when the layout of pages is unintentionally changed (MAGNOLIA-5029 -> MAGNOLIA-5132) or the dialogs were broken due to changes in javascript (MAGNOLIA-4829 -> MAGNOLIA-4895). Another problem is that the layout of pages between author preview and public are not the same (missing empty DIVs which anybody noticed - compare demo-project_author.png vs demo-project_public.png, empty optional areas rendered on public MAGNOLIA-5227) We should avoid this by adding tests for comparing sample screenshots from ce/ee bundles with current ones. This will also automate page rendering/layout QA.
What could be compared
- webpages (public/author + with/without page editor)
- samples module
- demo-project
- admincentral pages
- admincentral dialogs
Concept
What we can use for these tests
Selenium
Selenium is able to take screenshots already. All we need is to create pack of sample screenshots which will be compared with current ones.
- + Selenium is able to take screenshot of whole page even if it's not completely visible in browser (scrollbar).
- + We need to target ce/ee bundles to same location to avoid redundancy of sample screenshots
- + We have tested such comparison already (see MAGNOLIA-5216)
External tool
We could possibly use some external tool to do such comparison
- + Independency on bundles
- - Is anybody familiar with such tool already?
- - Does any tool take screenshot of page behind the visible screen as Selenium does?
Implementation
We have selenium tests in bundles already.
- 4.5 PageEditorTest uses Selenium1
- 5 UI tests use Selenium2
UI tests use taking of screenshots already and we didn't meet any issues when testing of taking screenshots with Selenium1.
How would be screenshots saved?
current screenshots
- Target folder: UI tests save screenshots into magnolia-integration-tests/tests/target/surefire-reports/ folder. We should respect this pattern.
- Screenshots names: UI tests name screenshots with this pattern:
ActivationUITest_activatePageResultInStatusChange_177_By.xpath%3A+%2F%2Fdiv%5Bcontains%28%40class%2C+%27item%27%29%5D%2F*%5B%40class+%3D+%27label%27+and+text%28%29+%3D+%27Pages%27%5D.png
This is insufficient for out task and we need folder structure to remain traceability in such big tree as demo-project website: magnolia-integration-tests/tests/target/surefire-reports/ComparingScreenShotsUITest/demo-project/about.png
We would save save probably only screenshots which would fail on comparison, others would be deleted after each comparison sample x current.
sample screenshots
in every bundle:- git db would be too big,
- update of samples would be not maintainable (4.5.x/5.0.x/5.x - ce/ee bundles).
- in separate module - only integration tests would have dependency
- locally on Hudson
- in separate git repository
Comparison:
Binary comparison is not usable, we cannot expect 100% match. Better would be pixel by pixel comparison. We can compute % of wrong pixels this way and set threshold to suitable value (5% seems OK, 5% threshold passes for small pixel-differences but is able to find missing empty DIVs - MAGNOLIA-5227). We should convert the images to black&wait to avoid problems with color shades (empty gray placeholders in have different shades of gray, at least when testing on 4.5 author instance). This would also prevent problems with color changes. We can check layout in B&W without any side effects.
Target URLs
The lists of URL for comparison should be saved in /src/test/resources/targetURls folder as files:
- website (website pages) as demo-project or Samples Module pages (/demo-project/members-area/registration/registration-confirmation)
- dialogs (optionally with parameters i.e. .magnolia/dialogs/useredit?mgnlPath=/system/anonymous&mgnlRepository=users&mgnlRichE=true&mgnlRichEPaste=textarea&mgnlCK=1375962974071)
- pages (admincentral pages i.e. .magnolia/pages/dmsZipUpload?repository=dms&path=/demo-project/img/logos/magnolia-logo&mgnlCK=1376047515166)
Browser dependency
Screenshots from different browser/OS are different Because it's not possible to force developers to use the same browser/browser version we have to:
make separate sample screenshots packs for different browsers (unmaintainable)- run these tests only on Hudson
Set of tests
Minimum
- compareWebsiteAuthorTest() - compare website pages from targetURLs/website list - author instance vs sample screenshots
- compareWebsitePublicTest() - compare website pages from targetURLs/website list - public instance vs sample screenshots
Additionally
- compareWebsiteAuthorVsPublic() - compare website pages from targetURLs/website list - author vs. public instance (no sample screenshots needed)
- compareDialogsAuthorTest() - compare admincentral dialogs from targetURLs/website list - author instance vs sample screenshots
- compareDialogsPublicTest() - compare admincentral dialogs from targetURLs/website list - public instance vs sample screenshots
- compareWebsiteAuthorVsPublic() - compare website pages from targetURLs/website list - author vs. public instance (no sample screenshots needed)
- compareAdmincentralAuthorTest() - compare admincentral pages from targetURLs/website list - author instance vs sample screenshots
- compareAdmincentralPublicTest() - compare admincentral pages from targetURLs/website list - public instance vs sample screenshots
Adding new tests & Maintenance
If the layout would change intentionally and this is not so often (we can always change threshold if the test would be too sensitive)...test would fail, failed screenshots will be in surefire/ directory on Hudson, we can check them visually and if everything would be OK, we can upload them as samples.
To add new target URL for comparison:
- upload new screenshot
- update or add new list of URLs to check