Context

Common requirements for high reliability UI testing

  • demonstrate product is usable in all browsers
  • demonstrate branding is OK in all browsers

Common requirements for QA UI testing

  • demonstrate product supports user stories/ scenarios
  • detect any weird crashes in obscure setups
  • ensure products are compliant with #a11y laws 1 2 3 4.

I am currently using Mocha as my primary test tool in JS. It can be run as BDD or a TDD, so there is less tool management.
Mocha can be run in browser 5 6. I have not found a extension like Swarm 7 for Mocha, but Jenkins 8 would work as a substitute. This article isn't about distributed testing, but Swarm seems like a good idea, good enough to be used for testing jQuery releases 9. For forking and parallel running, Jasmine is a better organised tool 10.

I did a quick literature search but didn't find anything useful. Obviously many people use Selenuim 11 to meet this requirement.

objective: use a library to concisely make declarative UI statements about webpages/ webapps (e.g. assert that all visible text is larger than 15px on screen). This MUST be run in a webrowser, in the normal operating condition. With this library, UI regression tests can be quickly defined. A unit test, in a well maintained library, can be easily ran in all browsers.

It is not intended to look at tag well-formed-ness, as that isn't UX/UI, but implementation. If tag well-formed-ness is required in a unit-test, I would recommend tidy 12.
I am not focussing on page load speed, as there are many tools that do this. It should be noted that there are many implementations / notations of features, and to be useful all of these need supporting. For example a font size may be specified in px, pt, em, rem, %, and more. The conversion will need to be done at runtime, as it is device and setting specific. A person reading the whole of this site may note the overlap between this tool and CAIN-project, cain-introduction.
Current JS file is available, requires tinyColor.js; which is in the same folder in GIT.

Metrics that are required (IMO)

  • font sizes
  • which font
  • color contrast
  • title/ tooltip is present & has content
  • image alt text is present & had content
  • check colour (used for branding)
  • current object location
  • literal text is present (This is a frail test, so I avoid these. Brand people care about this. Also absence of common spelling errors e.g. in product name)
  • text direction and alignment
  • absence of corrupt chars (see badly encoded UTF8, which shouldn't still occur)
  • language meta tagging (i.e. is X a page of Big5 text, and meta tagged as Big5, )
  • check if “on screen”/ get co-ordinates
  • detect visual overlaps
  • all 'label' elements are properly populated
  • HTML5 elements are present to support semantics and screen readers etc
  • content movement. As UX, I hate page re-flows, I don't need your advertising; I really don't want to wait for the advertising to load 6 different JS frameworks.
  • page load speed (as noted, use a different tool)

Method

  • font sizes
    • getComputedStyle, getPropertyValue
    • Convert all units 13 14
    • return value
  • which font
    • getComputedStyle, getPropertyValue
    • Results would need to be compared to a literal
    • There may be a algorithm to convert fonts to font-family 15 (don't know enough detail)
    • return value
  • color contrast
    • use check colour solution to get foreground and background
    • subtract one from other 16
    • return delta
  • title/ tooltip is present & has content
  • image alt text is present & had content
    • querySelector, getAttribute, check value length
    • return bool
  • check colour (used for branding)
    • getComputedStyle, getPropertyValue
    • Convert the units to a single standard 17 18
    • return value
  • current object location
    • Bounding boxes are part of HTML, see 19 20
    • return value array, need to make new array, as Browser one is RO
  • literal text is present (this is frail, so I avoid these)
    • Get all text, see 21, or select from supplied selector
    • Apply RegExp of argument to text block
    • return bool of match
  • text direction and alignment
    • Look for the attrib, as outlined 22, wrapping getComputedStyle, getPropertyValue
    • return value
  • absence of corrupt chars (see badly encoded UTF8, which shouldn't still occur)
    • huh?
  • language meta tagging (ie is X a page of Big5 text, and meta tagged as Big5, )
    • Extract attributes., according to 23 24. Country codes should be limited to 25
    • ?guess language of text? My initial plan involved use of RE code-point classes 26, but this isn't part of JS
    • can return first item
  • check if “on screen”
    • Apply current object location
    • return whether values are outside of a bounding box that is that browser size
  • detect visual overlaps
    • Takes two element ids
    • Apply current object location
    • Do maths to spot overlap
    • return bool
  • content movement
    • Must be sampled across time, do this last
    • Get location of item from passed selector, use location computation from other metrics.
    • Practically will needs to be run separately, as it will take the most time.
  • page load speed
    • blah, blah

My initial plan is to provide methods that take a HTML selector, and return the value of the named metric, in a normalised form. I am really fortunate that querySelector() is widely supported 27.
NB: tools must be made with older JS to supply higher reliability in older or awkward browsers, so no 'class' keyword.