Update AppVeyor image

[andreasjordan]

This will probably fail as it's the first try and I can not test this locally.

But let's try to update the AppVeyor image to "Visual Studio 2022" to have SQL Server 2017 and 2019.

[andreasjordan]

Now too many tests are still in the default senario. I can fix this by moving more tests to SINGLE or MULTIPLE, which is the plan anyway.

But before I do this: @potatoqualitee - Do we want to move to the new image and versions 2017 and 2019?

[andreasjordan]

Old output in AppVeyor:

[andreasjordan]

This is the new version:

There is only one yellow line left - I will take care of that later. So this way it is very easy to have a deeper look at the test run.

[niphlod]

not sure LESS lines by default is the correct direction. Nobody looks at the logs, the important bit is the result of the checks.
when things go awry, you need all the log you can get, and easily.

[andreasjordan]

We missed a lot of bugs because tests were green and nobody looked at all the warnings in the AppVeyor output. In my local tests, I capture all the warnings and fail my tests if the output contains warnings. So I would like to have a clean and readable output with green and cyan and only yellow if its important.
I mainly suppressed the info messages from install-dbatools-library.ps1 by adding a silent switch.

But anyway: We sure can discuss if all my changes go to in the right direction. And as always: We can pick only some of the changes.

I can not get this branch to pass all tests, because I would need to change some tests to go into the correct scenario. But if I change tests, only those tests run and you cannot tell if the changes to the infrastructure work.

So I will stop here and we can discuss the changes while I open another branch to move some more tests to the new scenarios.

[niphlod]

we kinda are saying the same things. Log doesn't matter, the only thing that matters is the test results. IMHO a verbose(r) log helps to identify root causes better than a slim(mer) one.
Tests failing and not making the actual "result" of the test fail is a problem of the test itself, not the runner and not the log.... right ?

[potatoqualitee]

gosh i see both sides but believe we can possibly make it verbose only when needed? maybe with an error log that doesnt infinitely export sometimes so it has to be disabled.

i always like older scenarios/versions. are we still covering 2008R2 sometimes?

[andreasjordan]

If you want to see the output of "install-dbatools-library.ps1" I can remove "-Silent" in line 38 of "appveyor.prep.ps1" (currently: & "$PSScriptRoot\..\.github\scripts\install-dbatools-library.ps1" -Silent).

If you want to still use 2008R2, then we can keep using the image "Previous Visual Studio 2015" instead of "Previous Visual Studio 2022". Then we can use 2008R2 for most of the tests. We could also use different images in different scenarios and then test some commands against 2008R2 AND 2019. But this will take longer as we test two times.

I can save my changes locally and revert part of the PR.

[andreasjordan]

But let's wait untill #10090 is merged and development is merged into this branch. Then I can revert some changes.

[andreasjordan]

About Testing against different versions: I thinks it is important to test all commands against different versions (including very old and very new versions) in regular intervals of about once a month. But AppVeyor is not the best place to do that. My idea is to have a lab with a lot of maschines with different SQL Server versions. Then we can run our tests in parallel against those instances. Like running all the test from the SIMPLE scenario in four parallel powershell sessions againt four different SQL Server instances with versions 2008R2 as the oldest supported by dbatools, 2019, 2022 and 2025 as the currtly supported by microsoft. An we can run the COPY scenario against pairs of versions like 2008R2 -> 2019 and 2019 -> 2025. we can then also run real HADR tests with two maschines having the same SQL Server version which is not possible in AppVeyor. And we can have an AD to test things like Kerberos.

So for AppVeyor we have to decide which tests should run on every push and merge against which versions. As we can not get the two most current versions 2022 and 2025, I don't care if we use 2008R2 or 2019. I mainly wanted to try this to see if I have understood how AppVeyor works. I can configure to use the versions you and the other contributors like to test against.

But I am a big fan of "use only code that is needed" and so I removed every line of code that was not relevant. And I'm a fan of "only show information that are usefull" and "don't show warnings that are not warnings" and so I removed some of the output. But I can put all that back in, no problem.

[niphlod]

on the "test on old versions" we might wanna think about our own "declared support" ... 2008R2 is out of extended support since 2019.
As for the "let's run tests seldomly on a custom lab":

• you really don't wanna find 1 month later that change in code resulted in a breakage
• who's gonna take care of that lab, with no oversight nor transparency/inspectability of the runs

Let's be honest: we either do CI or we don't. Only things ran at each and every commit and are in the open are there to last, everything else is to be considered as non-existant.

[andreasjordan]

I want CI tests and Lab tests. But we can not run all tests in CI because of time and infrastructure constraints.

Currently we run most of the commands against one version per command. That takes about 2 hours. If we want to test every command against an old and a new version, it would take 4 hours.

What is the maximum time, a CI test should take?

[niphlod]

we never ran a test on more than one instance. The reply to your question is another question: how much time do we want to spend to assure any test works in more than one instance ?

[potatoqualitee]

on the "test on old versions" we might wanna think about our own "declared support" ... 2008R2 is out of extended support since 2019.

No DBA wants to run software that is out of support but sometimes they must. It its extremely important to me to support people who are forced by their employers to run old versions of SQL Server and PowerShell. I remember working for an underfunded shop and having something like dbatools would've been my dream.

[niphlod]

been there but we're in 2026 ... we may wanna start saying "if it works it works, if it doesn't, it doesn't" for < 2012, IMHO.

[potatoqualitee]

2008 R2 SP3 seems like a reasonable compromise. I finally gave up my 2005 dreams 😅

[niphlod]

then it's by the grace of appveyor that we may assure 2008R2 SP3 compat.
as soon as the image goes away I don't see how the "we support 2008R2" statement can be sustained.

[andreasjordan]

Let me make a suggestion and see if we can agree on it:
We only want to use two instances to keep the environment as simple as possible.
All tests that require two instances will test against these two instances.
The tests that require HADR will test against the latest version.
All tests that only require one instance will test against the oldest version.
Then there are two options:
A: Image 2015 with SQL Server 2008R2 and SQL Server 2017
B: Image 2022 with SQL Server 2017 and SQL Server 2019
I don't care whether we use A or B.

We can later easily move between A and B, we just need something to start with.

[niphlod]

correction.
We may want to use just two versions to keep the environment as simple as possible, but nobody forces us to do so.
Testing e.g. 2008R2 + 2022 just makes sure command runs in 2008R2 AND 2022, not that it runs on all versions spanning FROM 2008R2 AND 2022.

We may want to run commands using a single instance splitting them on 2008R2, 2012, 2019 AND 2022. This won't ensure every command works on all versions, but it enlarges the surface area making sure SOME commands can run in more than two versions.
Having tests pointing to multiple instances for splitting scenarios in appveyor can be easily reduced "in bring your own lab" setting the same name for multiple variables (just as it was, albeit it wasn't properly documented which instances could "collide" and what instances couldn't be "the same").

At the end of the day, pending full runs which shouldn't happen if not very rarely (i.e. merges to master, touching things that are not public/*.ps1, etc) the current setup just runs the commands that are being changed within the commit, plus all their dependencies... the "2 hour" mark basically isn't reached for any meaningful "rapid development".

[andreasjordan]

https://www.appveyor.com/docs/windows-images-software/

Image 2015 has: 2008R2, 2012, 2014, 2016, 2017
image 2022 has: 2017, 2019

[andreasjordan]

We can use the "part" code to split all "InstanceSingle" across different versions while we can use the same version in local labs. I can implement this if we want.

[niphlod]

https://www.appveyor.com/docs/windows-images-software/

Image 2015 has: 2008R2, 2012, 2014, 2016, 2017 image 2022 has: 2017, 2019

this is a limitation only for commands needing MORE THAN ONE instance. Given "we care about old-ish", the only choice there is 2015.

For single-instance commands, any image is valid.... right ?

[andreasjordan]

By the way: We are not limited to use only one image. We can use a different image for every "matrix" entry. I have seen that in the docs.

[andreasjordan]

We can run our tests on any image provided. You choose the image and I change this PR to use that image. I personally don't care.

[andreasjordan]

New suggestion: We change as little as possible. So we use the old image and I try to configure the tests like they are today. And we can change that later if we like. we just keep the new naming (InstanceSingle, InstanceCopy1, ...) so that I can use the tests in other labs.

[niphlod]

By the way: We are not limited to use only one image. We can use a different image for every "matrix" entry. I have seen that in the docs.

I know, and I'm maying that much because I do know ... but also because I've seen multiple instances of "how bad is appveyor, let's do that instead" lately that maybe didn't take into account how much work it's needed not only to build a VM where to run tests (which is by itself a tad bit daunting) but to have an infrastructure readily available, inspectable, reproducible for everyone.

One of the reasons dbatools stands on its feet with most of the original maintainers leaving active development is IMHO the extensive (albeit it can always be improved) test suite. And with the introduction of a certain french coder its IMHO more important than ever.

I personally don't feel "the urge" to complete this PR without a proper "napkin-planned" decision of how we want to run the full suite ON APPVEYOR. Again IMHO (without disregarding the work done towards the goal), the fact that an entire suite can run in any "custom lab" (as in "bring your own", github runners which I feel are complicated to set up on a whole different level) is secondary, given said lab:

• is nowhere readily-reproducible
• isn't publicly available
• won't be running continously for each commit
• won't report its results back to github

[andreasjordan]

Ok, I see your point. I've changed this to a draft and we should not merge this.

Are there any changes in this PR that you, @niphlod, would like to see in the developmen branch? Then I would move them to a new PR.

And what changes to the current development branch would you like to see to move forward with the tests? Then I would work on that.

[niphlod]

I'd personally "sprinkle" single-instance commands to target 2016 and 2019, but if @potatoqualitee wants 2008R2 too I'd use that too.

Till appveyor offers something else then we'd have:

• multi-instances running on 2015 image with one 2008R2 and one 2017 (multi-instances should cover "migration-copy-whateverupgrade" scenarios and supporting such an "earlier source" should cover maybe 95% of the regressions)
• single-instance commands (let's say they are 300) that are split evenly:
  • 100 on 2008R2
  • 100 on 2016
  • 100 on 2019

And now, for the "probably unrealizable dreams":

• for "normal PRs" (i.e. targeting one or at most two functions that have single-instance tests) we should enlarge the surface targeting 2008R2 AND 2016 AND 2019. Taking 10 minutes or 15 don't really change that much in "development velocity"
• we should also test with both 5.1 and the latest PS Core

I'd also poll how many users are targeting Azure, AWS, GCP instead of on-prem because I feel in Jan 2026 the percentages shifted significally comparing to e.g. Jan 2020. Speaking of my own "report" I'm basically down to zero on-prem instances and 100ish between Azure Sql Database and Azure MI.

[andreasjordan]

I just tried to run most of the tests against the 2008R2 instance and a lot of them failed. 2008R2 is express, so all agent tests fail. Also all encryption tests. We would have to add the minimum version to all the tests (or the commands). I will now move to 2016 as the minimum.