Using Run Settings for integration and system testing
Getting configuration from the environment. What is the big idea? It is of course perfectly possible to code all the webdriver configuration into your tests as I have done so far, but what if I want someone else to be able to pull and run my code without having to go in and make code changes?
Maybe they have a grid set up on a different URL. Perhaps they want to run the same tests against diffferent browsers on different days or on dedicated machines. Before I go any further lets give some thought to what I want to achieve: There should be a default configuration that is used even if no configuation is specified.
There should be the ability to use standard xxxx. I have yet to work out how to use them in JetBrains Rider, but they can be set from the command line.
Three and four are to allow anyone to pull code from the repository and run it on their own machine, directing the RemoteWebDrivers to their own selenium grid, or forcing to a single local browser if preferred, e. Setting configuration using. The file appears on the Test Settings menu, and you can select or deselect it. While selected, the run settings file applies whenever you select Analyze Code Coverage.
I want two different settings files. The TestSessionTimeout allows me to prevent the test run from hanging if anything goes wrong. Accessing the configuration parameters in Nunit 3 tests From the Nunit 3 documentation for TestContext: TestParameters Test parameters may be supplied to a run in various ways, depending on the runner used.
For example, the console runner provides a command-line argument and v3. The static TestParameters property returns an object representing those passed-in parameters. The TestParameters object supports the following properties: Count — The number of parameters. Names — A collection of the names of the supplied parameters. In Vb, use Item. The TestParameters object supports the following methods: Exists string name — Returns true if a parameter of that name exists.
Get string name — Returns the same value as the indexer. Throws an exception if conversion fails. Note that all parameter values are strings.
You may convert them to other Types using the generic Get method listed above or using your own code. An exception may be thrown if the supplied value cannot be converted correctly. The following static method should suffice. Exists settingName? I decided to lower case compare against the default value for this.
In other words, if it exists but is not the default value set to! Exists settingName TestContext. Get settingName. Equals defaultValue. It also needs to be generic; in other words it needs to be able to return a value for any Enum class as I have three Enums in my WebDriverConfiguration object.
Writing generics is somewhat outside my comfort zone, but its all good practice so here goes: using System; using NUnit. Parse typeof T , TestContext. If you have any suggestions for improvement feel free to comment below or on Twitter. On re-reading this it sounds like I could have used the provided generic Get method and handled the exception. Oh well! I realised however that this then adds an addiional dependency on Nunit.
I want to keep the dependency list as short as possible and not imply that the WebDriverFactory is only useful for Nunit tests. Plan B then, create a new package. Again your feedback on this decision is welcome. Selenium; using static AlexanderOnTest. Utils; namespace AlexanderOnTest. And finally… Local file overrides Fortunately Newtonsoft.
Now I just need to find and read them. As ever I need to be mindful of cross platform issues here: File system separators vary between OSes, but fortunately System. Combine … works in an OS independent way.
For location: Environment. Combine fileLocationPath, filename ; if string. The value in an applied. Default Localhost grid.
Default values. Check a. By far the longest part of preparing this post! Lessons learnt: Cross platform testing is the toughest bit of. NET Standard library creation Testing complexity increases dramatically as the number of packages increases unsurprisingly!
The decision I made in the WebDriverManager implementation to store a delegate constructor rather than the requested WebDriverConfiguration made debugging my configs here much harder than it could have been. Get settingName to return a setting from a. GetFolderPath Environment. Combine folderPath, filename can be used to find the required file. A reminder: If you want to ask me a question, Twitter is undoubtedly the fastest place to get a response: My Username is AlexanderOnTest so I am easy to find.
My DMs are always open for questions, and I publicise my new blog posts there too. Next time: I will at last be exploring dependency injection using the native. NET Core functionality in the microsoft.
Override TestRunParameters in .NET Core
We have created infrastructure and a simple web service with automatic releases to production. How can we increase our confidence that anything that changes will not break everything? The answer is automated testing. The evolution of automated testing for many development teams generally requires much more transformation than teams expect, especially as most projects move to a more Agile process. In the past, most teams had a dedicated QA staff who tested the projects and confirmed the quality of a project.
It was time consuming, but fit into the long project schedules of months or years that the waterfall process was good for. When teams started to switch to Agile, it was logical that QA teams would automate these tests with UI testing tools such as Selenium. These test solutions were developed quickly, and became large and complex. Unfortunately, UI testing is a slow process, and these tests need many hours or days to run.
Enter the test pyramid. The idea is that functional tests should be used as a last resort, and the focus should be on unit tests. Unit tests are so self contained and fast, you can run thousands in a few minutes.
Azure DevOps itself runs tens of thousands of unit tests in a few minutes. Given this, and our test pyramid below, we have a few takeaways. When writing a test, we should attempt to write it as a unit test first. To achieve these testing goals in our project, we are going to build two new projects. For our unit and integration tests we will create a.
Net Core MSTest project, and for our functional tests. The tests project runs in our continuous integration process, giving us constant feedback in Visual Studio and feedback in pull requests before we merge with master. We also have functional tests we can run as a smoke test to confirm that each environment has been deployed without breaking something else. This is our initial test strategy. It will evolve, but will meet our initial needs. Creating unit and integration tests We create a new branch and open it in Visual Studio.
We are going to create basic tests against the ValuesController — the default controller created in the web service project. Using the test explorer, we run this test and it runs successfully and quickly -in just 19ms.
We are going to use Code coverage to help us confirm we have written tests for all of our custom code. We can run code coverage from the test explorer: We can see the code coverage results in the code coverage output window.
Looking at the results below, we can see that 22 of the 28 uncovered blocks are from generated code in the Program. We generally want to ignore generated files and only use code coverage for our custom files. This is a better indicator of what we need to write tests for — those 6 blocks in the ValuesController. Live unit testing is shown visually in the left column of our code in Visual Studio editor with the green checkmarks test running successfully , red crosses test failing , and blue dashes no test covering this code.
We write a quick purposely failing test to verify the Get id function, to help demonstrate this in the ValuesController. As the user types, live unit testing calculates what tests cover the edited code and reruns the affected tests to ensure that the test still passes.
This changes all of the live unit test icons to add a little clock icon to the bottom right corner of the icon. We correct the unit test error we made above, and the test now passes successfully. We will use this shortly, when we write our functional tests. Creating functional tests with Selenium Next we are going to write our Selenium functional tests. We add the following NuGet packages: Selenium.
WebDriver Selenium. This file helps to define our runtime properties. At the top of this class, we will add the using statement: using OpenQA. Chrome; Then we will add two private variables. The chrome driver and test context variables will be used to run our tests.
GetDirectoryName Assembly. Location , chromeOptions ; if TestContext. The attributes are important at the top. When running the tests locally, it will now use this file to configure the tests, and in this case, the web service url defined in this file. For the Selenium functional tests project we need some extra YAML to publish the project without zipping it, and to copy the chromedrive. This allows us to reconfigure the project configuration and run it on each environment easily.
Pushing those changes and examining the build summary, we are seeing a lot of detail now. This is a pull request build, so no deployment is created. The build artifacts contain the web service zip and the functional test files. All of the tests passed — but what happened to the code coverage? Looking closer at the code coverage, we can see the moq. Unfortunately, code coverage in. Net Core is still not quite complete yet. Fortunately, there is a workaround. If we had other external files affecting our code coverage we could add them here too.
DynamicCoverageDataCollector, Microsoft. Net Core is updated to fully support these scenarios. For example, if we add a feature and note that the code coverage has a significant drop, we can conclude that the testing for this feature needs further investment.
Adding testing to the release definion Now that we have unit testing as part of our build, we need to add smoke testing to our release.
We will deploy our service to the staging slot, and then run the functional tests on the staging slot. If the test passes correctly, we will swap the staging and production slots. In the past, we would have copied the files to the web server manually, tested the end point to ensure the service was functioning — for each of the three environments. This automation is an important step for our confidence in the continuous deployments, as we are beginning to add automated testing to ensure the service still works as expected.
Here is what the release looks like with the Smoke test task between the deployment task and the swap slots task With the release successful, every environment now has automated smoke tests! Wrapup Today we built a robust testing infrastructure that will serve our project well going forward. We created a unit and integration test project for our build, and a functional tests project with Selenium for our release. We looked at why the test pyramid is important and how code coverage and live unit testing can be used as useful testing tools in our DevOps toolkit.
Posts Tagged ‘runsettings’
Throws an exception if conversion fails. Note that all parameter values are strings. You may convert them to other Types using the generic Get method listed above or using your own code. An exception may be thrown if the supplied value cannot be converted correctly. The following static method should suffice.
Alexander on Test
Exists settingName? I decided to lower case compare against the default value for this. In other words, if it exists but is not the default value set to! Exists settingName TestContext. Get settingName. Equals defaultValue. It also needs to be generic; in other words it needs to be able to return a value for any Enum class as I have three Enums in my WebDriverConfiguration object.
Writing generics is somewhat outside my comfort zone, but its all good practice so here goes: using System; using NUnit. Parse typeof TTestContext.
If you have any suggestions for improvement feel free to comment below or on Twitter. On re-reading this it sounds like I could have used the provided generic Get method and handled the exception.
Oh well! I realised however that this then adds an addiional dependency on Nunit. I want to keep the dependency list as short as possible and not imply that the WebDriverFactory is only useful for Nunit tests.
How to change test naming in Visual Studio using NUnit
Plan B then, create a new package. Again your feedback on this decision is welcome. Selenium; using static AlexanderOnTest. Utils; namespace AlexanderOnTest. And finally… Local file overrides Fortunately Newtonsoft. Now I just need to find and read them. As ever I need to be mindful of cross platform issues here: File system separators vary between OSes, but fortunately System.
Combine … works in an OS independent way. For location: Environment. Combine fileLocationPath, filename ; if string. The value in an applied. Default Localhost grid. Default values.
MsTest vs NUnit: Which Should You Use And Why?
So you loose context. It will come out like this: Just imagine tests like this — it is not easy to separate them. Solution Now, NUnit3 has a concept of Test Names, which can be configured, either individually, or globally. It is done by adding a Test Name Template Pattern to the test. And as seen the Test Explorer now displays the tests with this combination. The next step is therefore to add a runsettings file and enable that. That done, you can add one out of 3 possible templates for the runsettings.
For this purpose, just choose and add the simplest one, which is the Parallell template. In the end of the file you find the NUnit settings. You have proper test naming for all your tests! Having a runsettings file present is a good thing anyway, because that is also used to control code coverage and a lot of other settings. Using it to set the test name pattern just solves one more issue, and makes your developer life a little bit easier!
Test Name patterns are ignored by the Live Unit Testing.