2024-07-16

File as parameter to PowerShell function

 Quite often I find it usefull to have a file as parameter to a PowerShell function. And most times I want to make sure the file exists when the function is called. In a PowerShell advanced function this can be done quick and short by a validation. More precisely by a ValidationScript attribute to the parameter.

A example on a function with a file as parameter with validation on existance could be something like

function Write-LogFile {
    <#
    .DESCRIPTION
    #>
    [CmdletBinding()]
    [OutputType([void])]
    Param(
        [Parameter(Mandatory=$true, ValueFromPipeLine=$true, HelpMessage='...')]
        [ValidateScript({$_.Exists})]
        [System.IO.FileInfo]$LogFile,

Running a call to the function with a file object that does not exist

[System.IO.FileInfo]$theLogFile = "$PSScriptRoot\log.txt"
Write-LogFile -LogFile $theLogFile

… will generate an error like

PS C:\Scripts> .\FileParameter.ps1
Write-LogFile: C:\Scripts\FileParameter.ps1:26
Line |
  26 |  Write-LogFile -LogFile $theLogFile …
     |                         ~~~~~~~~~~~
     | Cannot validate argument on parameter 'LogFile'. The "$_.Exists" validation script for the argument with value
     | "C:\Scripts\log.txt" did not return a result of True. Determine why
     | the validation script failed, and then try the command again.

There are many other ways to validate if a file exists, but I like to use a FileInfo object as it give the functionality without much custom code.

2024-07-03

Stored Procedure Guidelines (T-SQL)

Here are some personal guidelines on writing and using a stored procedure. They are mentioned in a somewhat random order. The list and each recommendation will most likely change over time with new experinces, feedback and other new knowledge.

SET NOCOUNT ON

… to avoid unnecessary activity and communication on row count.

SET XACT_ABORT ON

… to roll back most transactions on error. See details from Sommarskog (link) and Swart (link).

SET QUOTED_IDENTIFIER ON

… to make the code a little more robust and strict. Also some protection against injection. Still I would prefer to enclose object and other identifiers in square brackets. This I think makes the code more strict and easy to maintain with several developers and teams involved.

Use schema name

... on used tables. Put procedure in custom schema without data. Do not use the schemas dbo or sys as they are not in your control.

Use tables

... not functional objects such as views, other procedures and triggers. And only use functions in stored procedure with careful big-O considerations. This could only be worse with the use of nested views, procedures, functions, triggers and other functional objects.

Use sets

... not cursors or sub-selects as SQL Server is set-based. Cursors are fine when on DB2 or Oracle. Sub-selects are often seen on SAS.

Use functional name

... not sp_ or other Hungarian naming. It doesn´t really matter what type the object is.

Use sp_executesql

hellip; for dynamic SQL. Not EXECUTE. In general you should avoid dynamic SQL. This includes prepared statements.

Use specific columns

... only take data that are required. Network traffic is then reduced to the required amount.

Use order of columns

... as given in the tables. One table a time. Re-arranging the columns could add significant resource consumption on each call.

Use keys

... and indexes with care. Use index on foreign keys. Check for covering index. Index usage might change with change of data in used tables.

Use table variables

... not temporary tables. This will keeps activity in buffer and away from storage,

Be direct

... and avoid temporary cached or staged data. Do not use dirty reads!

Be short

... by short transactions. Avoid costly isolation levels such as Repeatble Read. Also you can save some network traffic on often used stored procedures by short names on the procedure and the parameters.

Use effective parameters

... and avoid parameter sniffing. Focus on parameters where the values change a lot. Convert these parameters to local variables when suitable.

Check values

... before usage. Especially parameter values! Do the value check inside the procedure before usage.

Use effective types

... and avoid custom types. Custom types will add resource consumption on resources. And in some situation increase blockings on metadata.

Use effective operators

... avoid costly operators such as LIKE. Be carefull with COALESCE, string operations and aggregated values. T-SQL is for finding and sorting data. Use other languages for complex calculations and text processing.

Use explicit type cast

... not indirect or implicit. And you are still somewhat in control.

Use EXISTS

... not COUNT. As this is more effective with T-SQL.

Use effective error handling

... by testing before exception handling with TRY - CATCH. Consider exception handling as last-line-of-defence after tests and checks. Even when using exception handling this should be kept inseide the procedure. Throwing a exception outside the procedure could be rather costly on the resource consumption.

Use MS_Description

... extended property. This helps development with Visual Studio, see SQLAdmin on Descriptions.

Document

... procerdure, each parameter and output/return value in code, detailed and with examples. Also when you are the one and only developer and user.

Set security explicit

... on procedure through custom database roles on Active Directory groups. Using default database roles is outside your control.

Use ALTER PROCEDURE

... not DROP - CREATE PROCEDURE. Security and other external configurations will be lost on each change.

Monitor

... usage and performance with DMVs, XEvents etc. See Monitoring Performance of Natively Compiled Stored Procedures.

Call procedure effective

... with CommandType.StoredProcedure and SqlParameter objects on the SqlCommandobject. Use careful crafted connection string with secure authentication (SSPI) like Kerberos.

Reference

Kimberly Tripp, SQLSkills: Building High Performance Stored Procedures.

 

Top 10 stored procedure performance tuning tips in SQL server

History

2026-01-02 : Section on XACT_ABORT added. 2024-07-03 : Post published.

2024-05-13

Buffer Pool scan

In a SQL Errorlog we saw a lot of events like this.

Buffer Pool scan took 10 seconds: database ID 6, command 'CHECKPOINT', operation 'FlushCache', scanned buffers 341183509, total iterated buffers 345933594, wait time 75 ms. See 'https://go.microsoft.com/fwlink/?linkid=2132602' for more information.

The refered article "Operations that trigger a buffer pool scan may run slowly on large-memory computers" states that

Prior to SQL Server 2022, there was no way to eliminate this problem.

But then another event came around where we looked at Indirect Checkpoint configuration. It turned out that the database originally was so old that the "Target Recovery Time" was 0 (zero).

When we changed this configuration to 60 then the Buffer Pool scan messages no longer showed up in the SQL Errorlog.

Another benefit was that spikes and average on Disk Write Latency were reduced significantly.

Kendra Little wrote some time ago a useful article (link) with good references and details when Target Recovery Interval changed default value.

2024-03-31

Disassembling a hard disk

Hard disks with spinning platters are going away very fast. During easter cleanup in my workshop I found a old 750 GB hard disk, and decided not only to take it to the local recycling station but to disassemple it to see how it it constructed. During the disassembly I took some photos with my phone - they are not near professional quality but they are still usefull with the comments.

The drive is a Seagate Barracuda ES.2 (data sheet) with a SATA 3Gb/s interface. It runs with 7200 RPM which made it rater usefull when it was released. I stopped using the drive when I really swithched to SSD in my workstation. There is nothing wrong with the drive - it just went obsolete to me.

To remove the cover there are at first six T9 screws. But hidden behind the sticker there are three more screws, also T9.
With the cover removed there is direct access to the inner components. Also this is the point of no return to use the drive again as the disk is now filled with dirt from the normal atmosphere.
On the backside of the hard disk the circut board is placed. It is fastned with T6 screws but none hidden. The connection from the board to the inner drive is through some contacts. The picture below shows two sets of contacts with one set of three contacts for the electrical engine spinning the platters and another set of four by five contacts for the head.
A closer look at the board shows chips from Seagate and Samsung. The external connections are direct from the board to the physical interfaces.
Under the board and a metallic sticker there is another hidden T9 screw. This holds the head.
The connection for the head is on a small board. But is quite easy pulled out and disconnected from the head.
The head unit movements are controlled with a coil on the back of the arms and two powerfull static magnets. The top magnet is only held in place by the magnetic force and can be removed with a small screwdriver.
The arm with heads and coil is one unit together with a small board. This head unit is removed without any tools.
A closer look at the head unit from another angle show a small board and the arms with the physical heads to transport electro-magnetic signals to and from the platters.
Going from buttom to top on the picture below it looks like the buttom arm only hold one head to communicate with the lower side of the lowest platter.
The next two arms holds two heads each to communicate with the upper side of the platter below the arm and the lower side of another platter placed above the arm.
The fourth arm hold only one head to communicate with the upper side of the top platter.
There is a fifth arm but this hold no heads and is not connected on the board.
I guess that the head unit was designed and partly constructed for up to four platters where this hard-drive only has three platters as shown later.
The bottom magnet that was under the head unit is also quite easy removed without any tools. I guess that the small plastic spring to the right on the picture below is some sort of anti-vibration or shock-absorbtion.
The spindle holding the platters on the axle is removed with six T9 screws.
The first platter is removed without tools. Between the platters there is a open ring of metal fastned to the chassis. I guess this is to stabilize the platters when they are spinning. Also there is a inne ring to give space for the heads.
After all three platters are removed the chassis looks quite empty. I have tried to lay out the components to show the stack from buttom to top. Between the platters there is a inner ring and a open outer ring.
The outher part of the electric motor spinning the platters is removed with brute force as indicated by the tools on the picture below. The outer part is the static magnetics where the inner part is the coils. How the outher part is fastned I can't see. There are no obvious traces of glue.
The three-pin connector for the motor on the back of the chassis is glued on but easy removed with a small screwdriver.
The inner part of the motor with the coils is removed with brute force and another set of tools.
All in all I count about 55 major components in the hard drive. In that there is about 31 screws.
Looking at modern storage there has certainly been some major changes. Compared to a NVMe storage unit with only one major components in a single board and no mechanics the technological span is amazing. But also compared to an elder SAS SSD drive typically with one board, around three screws and two metal half-shells the technological evolution is significant.

In general and looking back the flash storage was a paradigm shift where all our notes on spindles, RPMs and configuration tricks like short-strike suddenly were to no use. With NVMe many rules of thumb were obsolete. As PCIe (PCI-SIG) evolve we will see new fantastic features also on NVMe and CXL. But the next paradigm shift does not come from evole the current mainstream technology but when a completely new technology matures and push to the side the mainstream tech.

2024-02-03

PowerShell Advanced Function Template

I have made this simple template to start a new PowerShell script file with a advanced function. It helps me to get the basics right from the beginning. Also it supports my effort to write all scripts with advanced functions.

The code for the template on a advanced function is on SQLAdmin GitHub (link).

PowerShell script documentation

To PowerShell is defined a syntax for comment-based help in scripts (about_Comment_Based_Help). And there are many more keywords than I have used above, but my personal PowerShell style is covered with these keyword. Feel free to create your own script template with our own personal selection of standard keywords.
Using only standard keywords ensure that your scripts can be presented in a central place with nice and useful documentation. And the presentation can be automatic updated without redundant documentation tasks.

Michael Sorens has on SimpleTalk posted a series of great posts about automatic documentation on a library of PowerShell scripts. The first post is called "How To Document Your PowerShell Library" where the basics are presented in a brilliant way.

Advanced function

If a script is evolving to become a central part of a process then you should elevate it by refactoring it to an advanced function.
The template above is for an advanced function, but not all possibilities are included.  The documentation on advanced functions (about_Functions_Advanced) and parameters on advanced functions (about_Functions_Advanced_Parameters).

A very important part of an advanced function is the CmdletBinding function attribute (about_Functions_CmdletBindingAttribute) that will make a function work like a compiled CmdLet. You can in most cases just use it blank and it will make a function work really nice as a CmdLet, but if you take some time to read and experiment you can create some awesome advanced functions. If you are looking into very large amounts of data you should take a look of the possibilities with CmdletBinding. More technical CmdletBinding is a PowerShell accelerator of the .NET class CmdletBindingAttribute that is a part of the namespace System.Management.Automation.

One thing that you will benefit from rather quick is the use of -Verbose or -Debug where it will by passed on through advanced functions. Then you will get ectended information not only from your own code but also from real CmdLets and .NET objects.

There are several articles that covers various facets of advanced functions ok, but my personal reference is, beside the documentation, the book „PowerShell In Depth“ (e.g. chapter 32) by Don Jones (@concentrateddon), Richard Siddaway (@RSiddaway) and Jeffery Hicks (@JeffHicks). Look for latest edition.

PowerShell script module

When you have refined the script, I think it is time to move the functionality to one or more PowerShell script modules. The general concepts about PowerShell modules are listed in the article „Windows PowerShell Module Concepts“ and short described in the MSDN article „Understanding a Windows PowerShell Module“. Right now I will focus on a script module as this text is about PowerShel scripting. There is a short and nice introduction in the MSDN article „How to Write a PowerShell Script Module“.

In the script template above I have indicated how to dynamically import a script module placed in a given path. If the script module is not signed and the PowerShell Execution Policy is somehow restricted – which it should be! – you can't import a module placed outside the local computer, e.g. on a file share through a UNC path.

PowerShell script modules are defined and stored in psm1-files, but looks a lot like plain script files. If you use the script template above for a script module the module import and the final lines with invokation should be removed. The invokation should be in the script file importing the script module.

There are several benefits of using PowerShell script modules. Many of them you will see when you start using them...
One thing that surprised me was the value of unit testing the script code.

Michael Sorens (again) has on SimpleTalk written the great article „Further Down the Rabbit Hole: PowerShell Modules and Encapsulation“ on PowerShell modules and the surroundings with some very valuable guidelines.

History

2017-03-15 : Blog post created on SQLAdmin blog.
2017-05-17 : Section about script documentation added. Inspired by comment from my old friend and colleague Jakob Bindslet.
2017-06-14 : Section about PowerShell script module added.
2017-07-04 : Section about PowerShell Advance function added.
2017-11-10 : Blog post text migrated from SQLAdmin blog and code to Github.
2024-02-03 : Blog post migrated (back) to SQLAdmin. Code migrated to SQLAdmin Github.