@hdkarlsen Welcome. If you follow @feditips and @FediFollows you'll get lots of advice on the Fediverse. Also if you put lots of hashtags for your interests in your intro post you'll get more followers too.
#Psychology #Psychometrics

@TomJewell

On my way out, links to two texts that I think are very useful for researchers interested in exploring and applying some of these arguments to #HRQL research and practice:

Fit for purpose and modern validity theory in clinical outcomes assessment
https://rdcu.be/dgE92

Constructing arguments for the interpretation and use of patient-reported outcome measures in research
https://rdcu.be/dgFae

And with that I am on my way

@TomJewell

This is all not new and relates to fundamental ideas of #sampling in #Statistics, #Epidemiology, and #Norming in psychometric applications.

In this educational piece I co-wrote with Edwin de Beurs and @admin we refer to many of these principles and the importance of the sampling process -- and more importantly, many of the older references we pulled together are good places do start reading up on the historical treatment of this topic:
https://onlinelibrary.wiley.com/doi/10.1002/cpp.2742

@TomJewell

We have argued that the definition of the target population, #sampling procedure, and #ItemContent of an instrument jointly define the meaning of measures in particular applications (#PsychometricEpidemiology https://rdcu.be/dgE5g).

Written from the perspective that we were perplexed by psychologists being surprised that psychometric results change when population and sampling procedure are changed, I think it is an illustration of the above from the other side.

@TomJewell

There will be plenty of other examples, this is just my particular pocket and the time I could spend on this. There is some more...

5) _Sampling and #RangeRestriction_

Language in the "Standards" often points to range restriction as a reason for changes in the criteria.

This has particular consequences in the case of #FormativeMeasurement discussed in detail here:
https://journals.sagepub.com/doi/10.1177/1094428110378369

And w that we move to the combination of #sampling, #epidemiology, and #Psychometrics.

@TomJewell

I argue that there is some understanding in the #psychometrics and professional communities that use #TestScores that evidence of quality is tentative as any statistical result.

And in my personal view, the different methodologies #COSMIN has developed (e.g., https://link.springer.com/article/10.1007/s11136-018-1798-3) or is currently pushing forward (https://doi.org/10.1186/s13643-022-01994-5) are tools to aggregate available evidence exactly for this reason: each individual study and estimate offers an incomplete picture.

@TomJewell

... if the norming and quality data have not been evaluated for eight years in the target population, then at least a justification needs to be provided why they are still used (e.g., https://link.springer.com/book/10.1007/978-3-662-53772-5).

@TomJewell

Their "#Guidelines on Test Use" stress to
"Choose technically sound tests appropriate for the situation" and they offer a fairly extensive definition of what a "test" is, which includes settings, procedures, and other parameters to consider.

They also stress
"...evidence of reliability and validity for their intended purpose"
and that evidence should
"...support the inferences that may be drawn from the scores on the test".

https://www.intestcom.org/page/15

@TomJewell

To which degree) the term "validity" includes all quality criteria and qualities one can consider to describe the use of #TestScores is a matter of heated discussion in fields related to the philosophy of measurement. But it is probably safe to assume this for the stuff covered in the "Standards".

For a look into the discussion I recommend Kathleen Slaney's book on Construct Validity (https://link.springer.com/book/10.1057/978-1-137-38523-9) and several of her papers deal with this topic.

@TomJewell

3) _It's about #TestScores_

Not always discussed explicitly in relation to all quality indicators and most frequently in the relation to "#validity", the process of validation produces evidence for the validity of #TestScores (as opposed to the validity of the test!).

The "Standards" make that very clear for the term "Validity" [2014, p. 11]:
"...the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests"

@TomJewell ... [2014, p. 97] "Renorming may be required to maintain the validity of norm-referenced #TestScore interpretations."

Many more places in the "Standards" are likely related to the overall argument

But this should suffice to illustrate the idea that professional test use acknowledges that a psychometric indicator is estimated on a sample from a population and the same epistemological questions apply as to any other place where we use estimates from samples.

@TomJewell

2) _Professional Standards_
I looked into the AERA-APA-NCME "Standards" (2014)* whether this is considered in practice.

Standard [1.8] for example explicitly considers that "[s]tatistical findings can be influenced by factors affecting the sample" and discuss how evidence for validity may therefore differ by context.

[1.10] highlights that reporting should be detailed enough that applicants can judge relevance of results for their setting.

#Psychometrics
*https://en.wikipedia.org/wiki/Standards_for_Educational_and_Psychological_Testing