Features overview
#
What is Canarytrace?Canarytrace isn't a testing framework, but it's a plug-an-play stack for functional testing, web performance testing and availability monitoring web application by user perspective. It’s based on WDIO testing framework, featuring live logging services, browser network sniffing, web performance testing and other components for designing, execution, evaluation and investigation of the results.
Other components are responsible for alerting (based on predefined thresholds), automatically searching for incidents in the data collected, reporting, searching, and visualising data from each run.
- Look at Canarytrace Architecture
#
E2E Functional testingSmoke Pro- Full control over the web browser via WebdriverIO in a stable pattern 1:1:1. WebdriverIO is used as a main controller of a browser.
- Design test case is BDD style. E.g. it and describe method.
- We use WDIO API v6
- All test report data are continuously stored to Elasticsearch.
#
Web Performance Testing (WPT)Smoke ProCanarytrace Smoke ProCanarytrace collects many non-functional metrics such as FP
, FCP
, FMP
, ResponseTime
, TTFB
, SpeedScore
, WebVitals and more.
WPT is an important activity during functional web testing. If the browser doesn't have sufficient resources, the web application may not work well. Thanks to the Canarytrace architecture, measurements are stable, repeatable and the results reliable. All non-functional metrics data are continuously stored to Elasticsearch.
Look at our base Awesome Web Performance
#
Frontend analyserOther testing frameworks evaluate functional state of the web application without any information about the non-functional context and behaviour of the web browser. Canarytrace stores many non-functional data metrics from the browser for analysing the state of the web application. E.g. coverage audit, all requests, all responses, memory usage in the tab with tested web application etc.
Canarytrace doesn't test web application as a blackbox and that is a huge difference and a great benefit compared to other E2E GUI functional testing frameworks.
#
User Journey monitoringArchitecture of the Canarytrace meets the requirements for pattern 1:1:1 and thanks to that it is a ideal tool for testing and monitoring web application from user’s perspective. Do you need measure performance all steps from login to add to basket on your e-shop? No problem!
#
Hero ElementsEasily measure render time of individual parts of your web application with Hero Elements, e.g. when exactly is displayed to user login button or some banner?
#
Live loggingAll data from Canarytrace runner are continuously stored in Elasticsearch. Other testing frameworks usually store results from the execution in a database or generate HTML report.
First problem with HTML reports is, that design and values in a HTML report created some vendor of test framework and this can be restrictive. Second problem is storing a large number of html reports and searching in them. There is also no easy way how to generate trends from these reports, how to store and distribute them and exporting it in some different format to developer, architect or test manager.
Canarytrace uses Elasticsearch stack for multiple activities. e.g. search, aggregation, visualisation, generating test reports and so on.
#
Canarytrace runnerCanarytrace Professional runner is responsible only for cloning test cases from the repository, loading and executing test cases / monitor script and continuously storing data to elasticsearch (e.g testreport, metrics). That’s all. No alerting, no parallel execution etc.
Elasticsearch stack and Canarytrace Listener take over other activities like alerting based on thresholds, reporting to stakeholders and first stage analysis of data from Canarytrace runner.
#
Maintenance freeCanarytrace Smoke and Canarytrace Smoke Pro are editions without any maintenace. Enter only destination addresses of landing pages, that's all.
It’s a great approach for quickly monitoring and measure of landing pages and other pages available via URI
#
Live reporting test reportAll data from Canarytrace runner are continuously stored into Elasticsearch. You can see the results of testing and measurement immediately.
#
Canarytrace installerInstaller is our component for quickly setup Elasticsearch and Kibana before first run. Installer set up index patterns, mappings and template, visualizations and dashboards.
#
Kibana visualizations and dashboardsKibana is a web application with GUI for viewing data stored in Elasticsearch. Data can be used in graphs and other visualisations from which you can create custom dashboards for your testers, managers, developers or other team members.
Kibana contains input search for searching keywork in all data. You can search and pairing data in all indices for every run by label uuid
#
Filebeat loggingFilebeat continuously stores stdout and stderr stream for every container with Canarytrace components.
#
DockerizedAll Canarytrace components are dockerized, so they are ready to be used on Windows, Linux and MacOS.
#
Kubernetes and OpenshiftArchitecture of Canarytrace is designed for use in Kubernetes or Openshift. Thanks to this architecture is Canarytrace without vendor-lock.
#
GIT and cloning repositoryCloning git repository with monitor scripts directly to container with Canarytrace.
#
Canarytrace LifecycleStoring attachments to disc or to AWS S3 All attachments such as screenshots, HAR and videos are stored locally in the docker container or transferred to AWS S3.
#
Wait for load event endCanarytrace has implement advanced features that allow you to overcome unstable frontend issues. We know the internal processes of browser, therefore, we extend WDIO with more sophisticated functions e.g. browser.waitForloadEventEnd()
for wait on latest browser event or browser.waitForClickable()
for wait on active element and any more.
#
Waiting on network and mocking requestsCanarytrace can wait on more browser events e.g. waitOnIncreaseEntries
for wait for incoming response, or mocking selected or all requests on server. It's useful for debugging javascript behavior.
#
Performance entriesperformance.entries
is collection of a part of Web Api Performance and contains events of web page and browser with non-functional metrics and timestamps. Garant of this collections is a web browser and whole collection is stored into Elasticsearch.
#
Console InterceptConsole Intercept is Canarytrace service, which intercept all entries from browser console. Contains message from recommendation level to error level. Again, all data from browser console is stored into elasticsearch.
#
Coverage AuditWeb application load a lot of more unused resources such as javascript or css. Coverage Audit is collection unused bytes per resource, which is loaded to web application during parsing.
#
Memory InterceptWeb application use resources such as CPU and Memory for every opened tabs. One web application runs in one tab. If a web application consumes a large amount of memory in the browser, they may not work properly, because missing resources. Canarytrace coninuously collects information about used and available memory.
#
Lighthouse and Performance AuditEvery phase loading of a web application and behavior of a web browser fired event. Performance Audit is service, which collects these events for automatically analysis. Canarytrace collected e.g. First Contentful Paint
, Time To Interactive
, Speed Index
, Cumulative Layout Shift
, numTasksOver50ms
, throughput
, Time To First Byte
, ResponseTime
, performanceScore
, loadEventEnd
etc.
For Performance Audit we use Lighthouse, which allow set params for emulate capabilities of mobile and desktop devices and for example throughput.
#
Request InterceptRequest Intercept collect all requests trigered from browser to infrastructure.
#
Response InterceptResponse Intercept collect all responses incoming from infrastructure to web browser.
#
Canarytrace ListenerCanarytrace Listener is a separate component and main features are:
- Analyzes all data / all indices data stores in Elasticsearch.
- Triggering events by thresholds.
- Sending report to many services, e.g. slack.
- Creating summary reports.
- Analyzes headers responses and resources of web application.
- Observing a live reporting from Canarytrace runner.
- Do you find mistake or have any questions? Please create issue, thanks 👍
- Have more questions? Contact us.