While writing a book (Leanpub) it might be useful to quickly replace a lot of variables in one go. Therefore I created this tool. GenerateBook-all-1.0.jar
The tool expects to following directory structure:
|-orig/ (folder containing the original files)
|-manuscript/ (if this folder does not exist it will be created)
|prop.properties (file containing all variables you want to replace)
Example prop.properties file content:
authors=Author1 and Author2
To use this tool: java -jar GenerateBook-all-1.0.jar
Copied: orig/chapter1.md to manuscript/chapter1.md
I hope this tool is useful for you and additional features can be requested via comments.
My first year as a independent software testing professional. This year I worked as a Technical Test Specialist/Coach in the energy branch. My main task was/is to setup a testautomation framework and support 10+ scrum teams with creating automated testscripts.
I experienced/applied some new techniques, like: docker, vagrant, gatling, wiremock.
I initiated a interest group for technical testers. We periodically organize sessions to share knowledge about technical testing (tools / methedologies / practices).
I also achieved the following certificates:
- Professional Scrum Master (PSM 1)
- Professional Scrum Developer (PSD 1)
- Professional Scrum Product Owner (PSPO 1)
Next year I will continue my current assignment.
You might think that protractor is the way to go when testing AngularJS applications… The thing with AngularJS is that you can decide to remove all class and id‘s from the rendered html. (which is good for page rendering times, but bad for testability). In the template files there are some identifiers, but those are not visible in the rendered html, all the webelements with a template identifier have a class with the value ng-binding (or something very similar).
We have a very large broad IT-landscape with a big variety of applications. (JSF (JavaServer Faces) websites, AngularJS websites, JMX services, web services (REST and WSDL), mail server integration, PDF reader, FTP, Ora cle DB and perhaps mobile in the future).
In this case I decided to write a Java-based test automation solution. Protractor is built on top of WebDriverJS. So, I thought the same actions can be done in Java. I found https://github.com/paul-hammant/ngWebDriver which suits our needs. Don’t forget to set the setScriptTimeout(), otherwise it might not work.
People tend to use BDD-tools for everything those days… Don’t get me wrong behavior driven development is a great way of software design, but it’s not -just- a testing tool! It’s a way of collaboration and it aims to reduce misunderstandings.
Aslak Hellesøy, Founder Cucumber Ltd,wrote a very good blogpost about it, where he pointed out some very good points. (https://cucumber.pro/blog/2014/03/03/the-worlds-most-misunderstood-collaboration-tool.html)
When using BDD-tools you’re implementing several extra layers in your testautomation code, which you need to maintain. (Both Gherkin features and step definitions) So, if you use it just for specifying testscript then you introduce extra waste.
Like Aslak already wrote, it can be justified if it improves collaboration and reduces misunderstandings, but if the tool is of testing purpose only those benefits will obviously never happen.
I also like his point about many testers fallen in the same trap…
…they use Cucumber uniquely as a testing tool. No collaboration. No outside-in. They write Cucumber scenarios after the software instead of the other way around. Doing BDD well leads to decoupled software, which again makes it easy to reduce the duration of tests. You don’t get any of these benefits when you treat Cucumber as an afterthought. Personally I have seen many companies using BDD as an afterthought. Often they come up with the argument that ‘everybody can write the feature files’, however they don’t look at the long term effect. Because maintenance can be hell.
I hope you all read the blogpost of Aslak Hellesøy and decide to use BDD in a proper (collaborative) way or decide to not use BDD at all.
It still seems to be a very popular tool, just because things get very readable for non-techie people. They are still a lot of drawbacks: duplicates code, scenario’s written in plain text, extra abstraction layers, etc… JGiven tries to solve some of those drawbacks, as they state on there website:
- Scenarios are written in plain Java with a fluent Java API – Neither text files, nor Groovy is needed
- Method names are parsed at runtime and define the scenario text – No duplicate text in annotations is needed
- Scenarios are executed by either JUnit or TestNG – No extra test runner is needed, thus JGiven works with all existing IDEs and build tools for Java out-of-the-box
- Scenarios are composed of multiple, reusable so-called Stage classes – No more test code duplication
- Scenarios and steps can be parameterized for writing data-driven tests
- JGiven generates HTML reports that can be read and understand by domain experts and serve as a living documentation
So, try to use code driven BDD frameworks, if you have to use BDD-tooling, because you can apply all kinds of development practices. (… and it just makes it easier)
In this blogpost I like to discuss some of the software testing pitfalls I encountered over the last few years. Some of them are related to software testing in general, others are related to specifically test automation (Because that is my main area of interest).
This might be a never ending discussion. My personal opinion; certification is definitely not the guarantee for a good tester, but it allows people to talk a common language. Furthermore, it gives you some proven practices and methods. How much you can use of these methods and practices, depends on the context. I often saw people who became a tester with a background in the area of development or operations. It’s funny if you hear them talk about software testing. Meetings with people without a common knowledge are not efficient at all!
I mostly work in environments where we have to test an web application. I often hear tester say that they don’t see the value of applying testing techniques, because it’s just a web application. I think this is really odd, because why should you relate the use of testing techniques to the nature of the application under test (test object)? It makes more sense to relate the use of testing techniques to the business value of the application under test. You have to test more if the business value is high and testing techniques help to produce more efficient test cases.
You will see it more often that there is less time spent on the preparation and specification phase of software testing. It’s mostly coming from teams who are saying that they apply some form of Agile. I think, you should always define some sort of testplan and test cases or test charter, regardless of the development methodology you are using. The level of detail depends on the context where you’re working in.
One of the most heard problems in software testing is the lack of test environments or up-to-date test environments. Utilizing modern tools we can create and distribute test environments on the fly. We also often see that data across environments is totally different. Having those two problems it”s hard to automate testing.
Recently I heard something funny. In a company where they apply Agile, they said: unit testing is up to the developers, we shouldn’t even look at it. (To save time…)
Developers are not testers and (probably) never will be. So, we need to help them with writing proper unit tests. Think of positive and negative cases and even testing all possibilities of a decision. (Also known as: decision testing or multiple decision determination testing) I think, it should also be our task to check periodically the quality of the unit test and verify if they still use testing techniques.
These are just a couple of pitfalls in software testing.
Functional test automation is relatively slow and heavy in execution. Selenium scripts for example need to launch a browser, open the website and testdata needs to be present. We can speed up testing by running the test scripts in parallel on distributed machines. This incredibly speeds up the in-browser tests and quick and accurate feedback is given.
In the time that I worked for Spil Games we decided to set up a Virtual Desktop Infrastructure (VDI) for both manual testing and automated testing. It’s essentially a single blade with 2 x 6 Intel Xeon E5-2630 @ 2.3GHz, 1 TB of solid-state drives (SSD’s) and about 130 GB of memory. Performance was an issue in the past, that’s why we have chosen for solid-state drives (the response of the virtual machine is ‘almost’ instant).
Manual testing: We had a pool of preconfigured operating systems and browser versions. The Virtual Desktop Infrastructure will create a virtual machine on demand, every time somebody wants to access a specific machine. It’s made by using VMware View that allows graphics acceleration over the PCoIP (PC-over-IP) protocol which makes it possible to test everything you want to in a browser and it is close to optimal performance.
Automated testing: We had a dedicated pool of virtual machines with DHCP reserved IP-addresses with which we can establish a Selenium Grid. We used virtual machines with Windows 7 and 3 GB of memory and all the required browsers installed. Remote test execution of the Selenium scripts is, with the described setup, even faster than execution on the local machine.
Image above created with http://www.gliffy.com/
- Sauce Labs: Run automated Selenium tests in the cloud or manually test your site on any browser instantly. Videos, screenshots, and developer tools make debugging a snap.
- TestingBot: Cross browser testing with Selenium Online. Automatically test your website on various browsers with Selenium.
- BrowserStack: BrowserStack is a cross-browser testing tool, to test public websites and protected servers, on a cloud infrastructure of desktop and mobile browsers.
As a test consultant I visit different clients. I have seen a lot of projects from various natures using different programming languages. Mostly I see projects using PHP, Java or C#. As a consultant we need to be very flexible, so we can work in those different environments. This is actually the best part of working as a consultant; you can develop a very broad knowledge.
So, I decided to put some very basic test automation projects on my personal github, written in C# and Java. These projects contain the use of a dependency management tool, Selenium/WebDriver and the basic use of the Page Object Model.
Github repository: https://github.com/roydekleijn/webdriver_java_example
Github repository: https://github.com/roydekleijn/webdriver_csharp_example
Remember these projects show you the basic usage of Selenium/WebDriver and may help you to get started with test automation.
Oohooh oh, more often JMeter is used to create complex test suites to execute functional test scripts against a Web Service. It”s perfectly possible with JMeter, but is this the most efficient way? It has to has to do with the project scope and priority, I think. As stated on their website, JMeter is designed to test performance.
“Apache JMeter may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types.” JMeter website
In my view JMeter is not the best tool to write a bunch of functional test scripts for a Web Service (infrastructure). You will mostly end up with some unmaintainable throw-away test automation scripts, which took quite some effort to create. However, JMeter is one of the best performance testing tools.
You might want to perform some pre (insert data in a database) and post (assert complex XML / JSON responses) actions while executing functional test scripts. Here for, JMeter is not that handy and flexible. Better is to write your test scripts in a programming language, this allows you to do more complex things and create a maintainable test suite (using abstraction). You can apply programming principles (such as: DRY (Don’t Repeat Yourself), KISS (Keep it Simple, Stupid!), etc.) and you can create an abstraction layer (Each significant piece of functionality in a program should be implemented in just one place in the source code.).
I will put my ‘RESTful Web Service Automation Testing Framework’ publicly available on Github, if I find some spare time. (This will include detailed example)
Last week we experienced a little outage on our websites (We have website with a lot dependencies with external system and third-party content). It gave us the opportunity to take a closer look at our products while having production issue. One thing we noticed is that we don”t tell the end-user that we are having issues. We have learned that we have to built-in a more resilient user experience during issues.
We can simulate these kind of failures while testing over a proxy. BrowserMob Proxy can be controlled by a REST interface and has some great capabilities, such as blacklisting and whitelisting certain URL patterns, simulating various bandwidth and latency and controlling DNS and request timeouts.
As I said, BrowserMob Proxy is a REST tool. In order to perform REST calls we have to install cURL.
- Download cURL from http://curl.haxx.se/
- Extract the compressed file
- Put curl.exe in c:/Windows/ (to enable curl from the command prompt)
- sudo apt-get install curl
Or some other installation command which is related to the Linux distribution.
Starting the proxy
In the first command prompt
In the second command prompt
Set the proxy in browser
- Navigate to Control Panel -> Internet Options * Goto LAN settings in the Connections tab
- Thick Use a proxy server for you LAN
- Fill in the address: localhost
- Fill in the port: 9091 (the one returned from the curl command)
- Click on OK twice
Limit the connection speed
Perform the following curl command in the command prompt:
curl -X PUT -d "downstreamKbps=50" http://localhost:9090/proxy/9091/limit
Now you can visit the website with a low connection speed.
Blacklist third-party content
Perform the following curl command in the command prompt:
curl -X PUT -d "regex=http://example.com/*.*&status=404" http://localhost:9090/proxy/9091/blacklist
Now you can visit the website while blacklisting some content.
Check the BrowserMob Proxy readme (on Github) for all available API commands.
For my current assignment I was asked to test the performance of an AJAX-based web application, which was quite interesting. The customer connects different local authorities, so they use the same solution for their services and operations. Hosting and development are done by two different parties and they blame each other about the (bad) performance of the web application. The customer hired Polteq as an independent party to measure server response times as well as client-side rendering times.
Server response measurement
A requirement for this project was to make it as maintainable and transferable as possible, since the client is not too technical. This can be achieved by implementing ‘CSV data set config’ and ‘HTTP requests defaults’ elements in Jmeter.
One of the most important things of a performance test is the reporting facility. I was suggested by Martijn de Vrieze to use jmeter-plugins, with which we can easily generate sophisticated reports. Jmeter-plugins can easily create the following reports: response times vs. threads, response times over time, response latencies over time and transactions per second.
Client-side rendering measurement
Maven, TestNG and Selenium WebDriver were the tools of choice to measure the client-side rendering times. I built a command line tool which executes testscripts based on given command line arguments (browser, testscript, runs). The tool executes the test scenario in the browser and stores some measurements while running the scenario. Based on the measurements a graph is created after execution.
The fun thing was that I could put the server under load by the Jmeter scripts and then measure the client-side impact with my WebDriver tool. First testresults showed a badly performing system and a lot of 404’s.
I will put the censored client-side measurement code on my public github after finishing this project.