>> [Elliott Smith:] Good afternoon everybody. And welcome to our first ever NDACAN Webinar. I'm, for those who don't know me, I'm Elliott Smith, the Associate Director of NDACAN. And I'll be hosting today's session. The focus of today's Webinar is the NSCAW, the National Survey on Child and Adolescent Wellbeing. As all of you on the call are aware, two different NSCAW studies have been conducted. Data collection for the first one began in October of '99 and ran through five waves of data collection over a span of five or more years. And then the second NSCAW, known as NSCAW II began data collection a little over eight years later in February of 2008. NSCAW II consisted of three waves of data over a three year period. The waves become very important as we talk today. So, the purpose of today's Webinar is to share important information with those NSCAW restricted release users who'd like to compare estimates between the NSCAW I and the NSCAW II cohorts. To lead us through today's presentation we're fortunate to have with us three members of the NSCAW research team from RTI International. Paul Biemer, Distinguished Fellow in Statistics at RTI is one of the co-PI's on the NSCAW. And he's been responsible for the statistical methodology for both NSCAW cohorts. And he'll be starting us off today when we start the presentation. We also have Sara Wheeless, Senior Research Statistician at RTI. On NSCAW Sara's been responsible for wave calculations and non-response bias analyses, those kinds of things. And then finally we have Keith Smith, Program Manager at RTI. For the NSCAW project Keith oversees analysis support and data quality assurance and also data delivery to us here at the archive.
>> [Paul Biemer:] Hello everyone. This is Paul Biemer. I'm going to be sharing the presentation with Sara Wheeless and Keith Smith, as Elliott mentioned. This seminar is all about how we estimate the differences between NSCAW I and NSCAW II. So, that's what we're going to be focusing on. We're going to be talking about essentially two ways to do that. One is called the intersection approach and the other one is using calibration weights. And there's some advantages and disadvantages to each of those that I'll highlight as I go through this. So you should be seeing a slide now that says course outline. And basically I'm going to review with you some of the key aspects of the NSCAW I and NSCAW II designs. The important point there is that, you know, they're very, very similar. There are some differences. And those differences are very important when we go to estimating change between NSCAW I and NSCAW II. So I'll be talking about some of the issues that arise when we try to estimate between cohort differences. I'll talk about calibration weighting, some cautions using calibration weighting. And then at that point I'll turn it over to Sara, who will talk a little bit about analysis and what this means in terms of how one conducts analysis. And then Keith Smith will then be talking about some specific examples that he has run and be going through those with you. And then we'll have and Q and A at the end.
^M00:03:39
So, just to kind of remind you about the NSCAW I design. This was a nationally representative stratified two-stage sample where we selected 100 primary sampling units which were counties or groups of counties. And then we used the agencies, the child welfare agencies, within each of these counties or groups of counties to be able to select our children. So, all the children that we select are basically reported to the child protective services. They are reported. They may or may not have been substantiated. But those lists of reports is what provides the frames that we use in these 100 PSUs. When we went into the PSUs, we stratified the children by eight domains, which were a combination of their age, whether or not they were in foster care, type of abuse and so forth. Also, the age was restricted from birth to 14 years in the NSCAW I. And that's kind of important because, as we'll see, it's a little different in the NSCAW II. We broadened the age range. And then there were four states that represent eight PSUs. So eight out of 100 PSUs refused to participate. And they cited what we call agency first contact reasons. What this means, basically, is that these particular states have laws in place where they would not be able to hand over the list of children to us as we could get in other states without first contacting the caregiver of these children and getting permission to do so. What this meant was that we were -- and we did some testing on this. We were getting very, very, low cooperation rates from that kind of process. So, they had to opt in in order to be able for us to contact them. And it just wasn't feasible to do the survey in those. So, we basically said they were refusing. They weren't really refusing. They just refused to participate in the way that the other 92 did. And because of the way they wanted to participate, we wouldn't be getting any data. So, we had to exclude them. In the NSCAW II, we tried to, as much as we could, keep the designs the same because we anticipated that there would be this desire to make comparisons between NSCAW I and NSCAW II. And to facilitate these comparisons, one of the things we did is we defined the target population exactly the same way with an exception or two to be noted in a minute. We used the same primary sampling units. So the same 100 PSUs that were used for NSCAW I we used for NSCAW II. We went back to the same ones that even those that refused before and asked them if they would participate this time. So, that helps when you're making comparisons because we're in the same areas. And so one can take advantage of the correlation between the estimates from NSCAW I and NSCAW II at the primary sampling unit level. We used equivalent sampling methodology. As we'll see in a minute, we used fewer domains than we did. Instead of eight, we used five domains in the NSCAW II. Otherwise it was very equivalent. We tried to use the same interviewing protocols, respondent selection rules, non-response conversion mechanisms. We used the same questions to the extent we could. There were some question differences, and that's, again, it's one of the cautions we have later on is be sure that when you do these comparisons that you're comparing the same characteristics of the same questions. Because there were slight changes to some of the questions, just in the spirit of improvement. Comparable weighting. Post-survey weighting adjustments and so forth. Now, the NSCAW II sample design used an age range of zero to 17 1/2. So, there is that one difference between NSCAW I and NSCAW II. That is, we have these older children now in NSCAW II. Because there was interest in that 14 to 17 1/2 group. I mentioned the five domains versus the eight. That doesn't really affect you guys at all in terms of making these comparisons. It does affect the weighting somewhat. But that's not an important issue for us. What is important is the fact that we have, in addition to the eight PSUs that where an agency first contact in NSCAW I we had four new agencies who also had passed legislation in the interim who said they also needed to contact any of these caregivers before turning over the list. And so those also fell into these agency first contact areas. So, that means that the areas of the country that were represented by these nine additional PSUs that had the agency first contact restrictions is a difference in the target population. So, they were excluded. They were excluded from the NSCAW II. And so therein lies one of the differences between them. And then we have the first bullet here, the age differences. So, one can say, then, these target populations are not identical between NSCAW I and NSCAW II. And this slide tries to summarize the key differences that we're going to be focusing on. The first is this age difference, zero to 14 versus zero to 17 1/2. And then the next is these eight agencies that opted out in NSCAW I. And then there were eight plus nine, or 17, all together that opted out in NSCAW II. In the NSCAW I, we computed what proportion of the child welfare population was represented by those who did cooperate, the 92 that did. And it's about 95%. In the NSCAW II, it's only 90%. So it's still a large proportion of the child welfare population. But you can see that if I were going to make comparisons between NSCAW I and NSCAW II, well, the first thing I'd have to do is I'd have to drop the children who were 14 to 17 1/2 because they aren't in the NSCAW I. And I'd be confounding that difference there. But the other thing is that I'd have to do something about the fact that the PSUs that are in the NSCAW II represent a slightly different population than those in NSCAW I. There's a considerable overlap, but there's still a difference in the populations that needs to be accounted for.
^M00:10:49
So this is kind of to show you sort of graphically what we're talking about there. So if I -- let me just back up one slide. If I start with this representation of the NSCAW I population. So that's what the 92 PSUs represent that cooperated in NSCAW I. And then I overlay that with the corresponding population represented by those PSUs that cooperated with NSCAW II, you can see that there's this difference up here for the children that are in that extra age group that we added on the NSCAW II. And there's a group down here in NSCAW I that isn't represented in NSCAW II because these represent the nine additional PSUs that opted out in NSCAW II. So, what does one do about that? Well, one easy thing to do is to try to just represent the group in the middle, where both surveys overlap. And so, we can do that by first of all removing the children here that are not in the NSCAW I. And you can remove these children here that are represented by these PSUs in NSCAW I that aren't in NSCAW II, right? So when you make an estimate of NSCAW I, you drop the children in these PSUs. And you also drop these children who are in NSCAW II when you make these comparisons. Now, what that does is it starts to restrict your population, you see? You're going to have to throw away data here. There's nothing we can do about the fact that the 14- to 17 1/2-year-old children are not in NSCAW I. We can't go back and get them, you know, eight, nine years later. But, there is something we can try to do to try to expand the inference from just this highlighted area in the middle, the overlap, to include this group here in the NSCAW I. And that's called calibration weighting. And I'll say a little bit about calibration weighting just to get the idea what we do there. So, let's talk about these two options. The first one being just the intersection approach. So, what that means, basically, is we're just going to develop estimates that represent this yellow region here. And that's called the intersection population.
^M00:13:23
So, I think I already mentioned that what we're going to be doing, then, is dropping the zero -- or we're keeping the zero to 4 in the NSCAW II, eliminating the 14 and above. And we're dropping the areas that are not in NSCAW II from NSCAW I. And so then what that gives us is an inference to the region that is highlighted in blue here. And as I mentioned, that's about 90% of the child welfare population between the ages of zero and 14. So, this does not include the older children. It does not include these PSUs, the children represented in these PSUs that are in the NSCAW I that didn't cooperate in NSCAW II. And that's simple to do because all you do is just basically remove these data from NSCAW I. Remove these data from NSCAW II. And then you make an estimate for NSCAW I, then make an estimate for NSCAW II. And now you can compare them. And what you're doing is you're comparing those two estimates on the basis of this population here, which -- it might be difficult to describe because you can say it represents zero to 14-year-olds. But then what part of the child welfare population is that that doesn't include the states that have the first contact? So that's one of the difficulties with using that approach. But it's simple. You don't need any special weights. You can use the NSCAW I weights that are available for whatever wave you're using. Use the NSCAW II weights for whatever wave you're using there. And you just form those estimates for that yellow region and away you go and make your comparisons. Now, a little more complicated approach is this option, which is using the calibration weights. It expands the inference to that group of children represented by the PSUs that were agency first contact in NSCAW II.
^M00:15:29
These weights are still restricted to zero 14. In other words, these weights are weights that we developed for the NSCAW II population. We gave the weights the value zero for children who are 14 and above, or above 14 years of age. So, if you use these weights, you're automatically going to be excluding the 14 to 17 1/2. But, the nice thing about it is that it has this coverage adjustment that expands the inference to include the agencies that cooperated in NSCAW I and didn't in NSCAW II. So it's like a coverage adjustment. It's a little more sophisticated than a typical coverage adjustment because of the way that we actually adjusted these weights. We experimented with a number of different ways, and this way seemed to work the very best and gave very good results. So, now using calibration weights, rather than restricting inference just to this yellow region that was the overlap, we now can expand to the NSCAW I population and the agency first contact states that cooperated in NSCAW I but did not cooperate in NSCAW II. So how did we do that? So the next slides are just going to talk about what the calibration weight is. And you really don't need to know any of these details just to use the weights. But, in case you're asked, you can give some explanation. And there's more detailed explanation in the documentation that we've written to describe how these calibration weights were actually computed. But, what I'm going to be doing here is just sort of talking generally about how to do it just to give you kind of the process in a nutshell. So, let's start again with this diagram we have. And you know, you can divide the NSCAW I population into two parts. There is this part that's in the overlap. And then this part that's outside of the overlap. And this part, P2, is representing agencies who cooperated in --- children under the agencies who cooperated in NSCAW I but didn't in NSCAW II. And P1 is the ones that cooperated in both and also zero to 14 years of age. Now what we do to do the calibration is we define a domain. And we use hundreds of domains. Because we use the five NSCAW II domains in each PSU crossed by the PSU. So theoretically you could have like 500 different domains. So we have lots of domains that we use. And we use a calibration model, which basically means what we're going to be doing is we're going to be trying to adjust the estimates based upon this domain, which spans both P1 and P2 to be equal to the estimates that are only in the yellow part of this circle in this domain. I'll talk about that in some detail now. So, what we want to do, then, is we start with the NSCAW I. So, we develop these adjustment factors, call them F sub D for the domain D. And we develop these adjustment factors using just the NSCAW I data. Because in the NSCAW I data, we know the characteristics of the children in the domain for both P1 and P2. So we know the children in this intersection. And we know the children in these intersections. And so what we can do is we can form two sums. We can form a sum that is just for the children that are in P1 intersect D. And we can form a sum that are in the entire D, both the P1 and P2, the union of those two. And what we do is we try to -- we come up with a factor here, F, such that when I multiply the weights by F, the sum for this region that's in the intersection of P1 and D equals the sum in the entire domain D, the P1 union P2. So, basically, you do that for, you know, like 500 domains, 500 of these special areas that we define. So you have lots of areas to make adjustments for. And you come up with these adjustment factors. And all of this is based upon the NSCAW I data. And then the next thing to do is then to take these adjustment factors and apply them to the NSCAW II data. And by applying them to the NSCAW II data, then we can adjust at these very, very fine detailed level, adjust the weights in the NSCAW II by these factors, F, so that in fact we attempt, then, to compensate for those children who are in P2. You know, we didn't get to measure them because they were excluded from NSCAW II. But we can compensate for them using these adjustment factors that we compute from the NSCAW I data. And that's essentially how these calibration weights are used. So, in the end, then, we get an NSCAW II weight that has as adjustment, which compensates for those missing agency first contact PSUs. And then we apply another level of adjustment here to, like another post-stratification adjustment, so that the weights will sum up to the NDACAN known totals.
^M00:21:26
Now, so what you end up with, then, is a set of weights for the NSCAW II which have been adjusted for the missing part, that P2. That is the agency first contact PSUs that didn't cooperate in NSCAW II, but did in NSCAW I. Now let me give you some advantages. There's advantages and disadvantages to both of these approaches. So when you use the intersection approach, it's very simple. You just lop off part of the sample in the NSCAW I. You lop off the 14 to 17 1/2-year-olds in NSCAW II. And then you make estimates on that reduced population and make any comparisons you want. So, one advantage is consistent. You're just using NSCAW II weights that are already available. You don't have to use any calibration weights, just use the regular NSCAW II weights and NSCAW I weights. The target population is the same as the NSCAW II target population. So it doesn't include that missing piece that was NSCAW I. It also excludes the 14 plus-age children. But otherwise it's the NSCAW II population as -- you know, if you were going to be doing an analysis of NSCAW II, you would use the NSCAW II weights. And that's the target population you would have inference to. It would be the same in this situation. Again, if you don't include the 14 plus-age children. It provides unbiased estimators. So there's no assumptions about the coverage adjustments that work or coverage biases has been adjusted for as you do in a calibration weights. And it's fairly easy to explain and understand because we're not using any special weights. We're just using the regular weights. The disadvantage is it limits the inference because that group that we eliminate from the NSCAW I constitutes about 5% of the child welfare population between zero and 14. So you're basically down to 90 instead of having 95 when you use the calibration weights. Your sample sizes will be reduced on the NSCAW I side because you're throwing away the data in those agency first contact areas. And also, because the weights have not been post-stratified to the intersection population, which add a little stability, these weights, if you just use the NSCAW II weights, have some instability as a result. It's like a domain estimate. Now, one thing we did is we experimented with something called intersection weights, which actually applies a post-stratification to those weights in that overlap population. And that adds a little more stability. But those really aren't going to be circulated, from what I understand, because these calibration weights are much superior. And you have another approach, if you want to use the intersection approach, you have another way of doing that without introducing a whole new set of weights. So, the advantage, then, of the calibration weights over the intersection approach is that, as I mentioned, expands to the 95% of the child welfare population. Uses all the NSCAW I data. You don't have to throw any data out for the NSCAW I. You still have to throw away the 14 to 17 1/2 on the NSCAW II, but you keep the entire NSCAW I data because, again, these calibration weights adjust the NSCAW II sample for that missing piece. And we also found that in a lot of analysis we were able to do that the bias is small when you do this calibration adjustment. It makes the assumption that the adjustment to all the calibration equation -- that these adjustments that we, these F sub D's that we obtain from the NSCAW I would solve similar calibration equations for the NSCAW II. So that's kind of the assumption that one makes with that.
^M00:25:36
Excuse me. I have a bit of a cold today. The disadvantage of using calibration weights is that -- it's not really a disadvantage, but the weights are zero for children age 14 or plus in the NSCAW II. Now, that's not a disadvantage if you're going to be doing comparisons with NSCAW I because you'd have to delete those children anyway, because they're not in the NSCAW I population. But, one -- I mean, I would advise against using the calibration weights for just a straight up NSCAW II analysis without doing this comparison with NSCAW I. If you were interested in these older children, the 14 and above, then obviously those weights would give them away to zero. They wouldn't be in any part of your analysis. But if you're going to restrict your analysis where you're not interested in those children, 14 and above, you could certainly use the calibration weights for all of your NSCAW II analysis, not just any comparisons with NSCAW I. And it would give you the advantage of having this inference to the larger population of agencies, the 95% that we keep talking about for the NSCAW I. But they're complicated to explain. You might agree with that after hearing my explanation if you didn't understand very much of what I was saying. But hopefully you understand enough to realize that, you know, there are some differences between the straight up kind of coverage adjustment that [inaudible]. And if you were going to use both the NSCAW II weights and the NSCAW -- the standard weights for NSCAW II and these calibration weights, you could get some small differences. They're not going to be large 99% of the time. But there could be some small differences in the estimates. So, again, you know, these were designed, these calibration weights were designed specifically for making estimates between NSCAW I and NSCAW II. And so you may want to just use those calibration weights when you do comparisons between the old and new cohorts and not try to use them beyond that. But that's up to you. They certainly are valid for a lot of analysis one would do with just the NSCAW II.
^M00:27:52
Calibration weights apply only to NSCAW II and are wave specific. We have them for wave 1 and we have them for wave 2. And we are also preparing them for wave 3. So there's some slight differences in the way we compute those calibration weights when you go from wave 1 to 2 and 3. So, so far we've only developed two of those waves. I've already mentioned this, but I think we can kind of skip that. The only thing I haven't really talked about here is this last bullet, which is a caution that I mentioned at the beginning of my presentation, which is, you know, weighting aside, one needs to pay attention to, when you're looking at a question item from NSCAW I and NSCAW II and you want to compare them and see, you know, how much has the child welfare population changed over the last, you know, nine, ten years? You may find differences in the question text. Response categories are different. Maybe the reference periods might be different. We tried to minimize that, but there are some instances of that. So we would caution that you just make sure and check the questionnaire items to make sure that they are comparable, or at least that you know the differences and can take into account any differences in the item when one interprets differences from using the calibration weights. At this point, I'm done with what I wanted to say. And I'll pass the ball over to -- or I guess Elliott will.
>> [Elliott Smith:] Yes, I'm passing it over now. Okay, Sara. You're on.
>> [Sara Wheeless:] Okay, right. I'm just going to talk generally about how you would use these with the software, various types of software. How you would use the weights and the various strata and PSUs that we've constructed. And then when I finish that, Keith will show some examples with SUDAAN. One example that we're showing here is if you wanted to estimate the change between NSCAW I and NSCAW II for a child's CBCL for some age group. We let y1 denote y1 bar be the child's CBCL for NSCAW I from wave 1 and y2 the estimate from NSCAW II, wave 2. We want to test whether there's any difference. We wouldn't really want to use just a regular two sample t-test because the covariance is not zero. By design we re-used or the PSUs are the same in NSCAW I and NSCAW II. So the covariance is not zero. So we can't just run a straight old t-test and be done. So, the proper way to proceed is we would concatenate the two data sets, the NSCAW I dataset and the NSCAW II dataset. And use either a procedure like Proct descript in SUDAAN or one of the SAS survey procedures. We could also use a regression, which is what this model shows, where our dependent variable is the variable we're interested in. And then we have a covariant. And then we also have an indicator variable for the cohort, whether it's a 1 or 2 for NSCAW I or NSCAW II. And testing whether the difference in the means is zero is equivalent to testing whether the cohort indicator is zero.
^M00:31:37
The weight variables that we have available or will soon have available, we have one for baseline or wave 1 comparisons, which is this N1N2CWT1 variable. We also have a variable for testing, or for making comparisons between the 18 month follow-up. The 18 month follow-up for NSCAW I was wave 3. And for NSCAW II it was wave 2. So that's one thing you need to keep in mind about these comparisons. And then we will soon have a third weight kind of finished, but we need to do some testing on it and maybe a few tweaks. But we also have a third weight -- or will have a third weight for 36 month follow-up comparisons, which would be the NSCAW I wave 4 data compared to the NSCAW II wave 3 data.
^M00:32:35
We also have, on the dataset with the weights, we have variables for the PSU. And stratum variables pseudo-stratus, pseudo-replicate variables, whatever your software calls it, cluster variables. The variables are the comp underscore STR and comp underscore PSU that denote the variance estimation strata and PSU. And we have defined comp PSU so that the PSUs that are the same in the two surveys will take the same values so the software will know the [inaudible] and will correctly estimate the covariance.
^M00:33:19
You don't really need anything different for software than what you've been using for NSCAW I or NSCAW II data. The software, just like what we did with the regular NSCAW I analysis or NSCAW II analysis, needs to account for the stratification, the clustering and the unequal weighting. And so that would be the SUDAAN, the SPSS Complex samples, Stata with the survey set option or the SAS survey procedures. There's probably some more out there, but these are probably some of the major ones. And the documentation or the user guide that we provided -- that is being provided with the calibration weights gives examples, I think, of SUDAAN, SPSS and Stata. So there are examples of different kinds of analysis, currently with the wave 1 weights with some hints on how to adapt it for the 18 month follow-up weights.
^E00:34:16
^B00:34:21
So, to make this work with the software, as I mentioned before, you need to concatenate or stack the data from the two surveys. So the data file that you run on will have to have data from both surveys in it. You'll and indicator variable that will denote the cohort or the survey. And we actually provided a variable on the weights file called Cohort, which takes the value 1 if it's NSCAW I data or 2 if it's NSCAW II. The analysis variables from the two surveys need to be named identically. So there may need to be some renaming if you're using a wave 4 versus a wave 3 kind of analysis. And I think Keith is going to show an example of that in his presentation. Then you just have to tell the software the name of the weight variable on whatever statement or option the software uses. And also tell the software the names of the variables for the variance estimation. And this is an example of SUDAAN code. In SUDAAN, we have a nest statement. So on the nest statement you would put the comp underscore STR and the comp underscore PSU. And down in the weight statement, in this example we're using the wave 1 comparison weight. So, the weight statement has the N1N2CWT1. And the dataset that we're running on is, like I said, a combined NSCAW I, NSCAW II variable. So the model not only has the dependent variable and the independent variable, but it also has this S variable, which is the cohort indicator. So that when we do any testing between the two surveys, you're actually looking at what's going on with the S variable.
^E00:36:10
^B00:36:16
Then other things you could use these weights for, you can test if two domains within NSCAW II are the same. Of course, your analysis would be restricted to the under 15 group. And you could also test whether, for specific domains, between NSCAW I and NSCAW II. So it doesn't have to be just the total population. You could test it for either the sampling domains or other domains such as race or gender or whatever else, other domains you might be interested in.
^M00:36:54
Now, Keith is going to show some real examples with -- run some examples with software.
>> [Elliott Smith:] Yes. And I'll hand control over to Keith. Thank you, Sara.
>> [Sara Wheeless:] Okay, good.
>> [Keith Smith:] Okay, great. So everybody can see the SAS code on their screen. Yeah, as Sara said, I'm just going to discuss sort of the mechanics of using the calibration weights. We'll use -- well, we'll be walking through some sample SAS code for creating an analysis file. And then some example SUDAAN code for generating the comparison output. And as Sara mentioned, there are similar examples for SPS and Stata in the appendix of the calibration and weights documentation. So, it's -- you start out by creating a dataset for the variables that you want to include in your NSCAW I analysis file. And so, here I'm just using a proc sort statement with a keep statement indicating the NSCAW I variables that we're going to be including. The next step is to sort -- and I'm sorry -- in both of these steps you want to make sure you sort by the child identifier, which in our case is the NSCAW ID. Then you create your file that includes the calibration weights. And again, sorting by NSCAW ID. Do the same thing for the NSCAW II, creating your data file of the survey variables that you want to keep in your analysis file. And then sort by NSCAW ID. And then the same thing for the NSCAW II calibration weights.
^M00:39:18
The next step is to merge the survey data with the calibration weights for each cohort, which is what these two data steps are doing. So you're merging the NSCAW I survey data with the NSCAW I calibration data. And you're merging it by NSCAW ID. And doing the same thing for the NSCAW II data. The next step, as Sara mentioned in discussing the analysis file that you'll need to create is to stack the two data files from the two cohorts. So, this is simply what this data step is doing. It's stacking the NSCAW I data file and the NSCAW II data file so that we have one file that has records from both NSCAW I and NSCAW II. And then, just some examples here. You can, you know, manipulate some variables as you would normally if you want to. In this example, we're going to be creating a re-coded service fee variable called RSERVC where we're just simply creating values of 0 and 1 instead of 1 and 2. And then for one of my variables that I'm going to be using in the analysis, I'm going to be setting the negative values of that variable to missing. So, in that data step, I stacked the two data files from two cards and then do any other type of re-coding that you want to do on the variables that will be included. So, let's just go ahead and run the code.
^E00:41:25
^B00:41:30
Okay. And then to check my re-coding of the service fee, I've got a little cross tab that you can make sure that the values are being re-coded correctly. And then the last step before you can do any analysis is to sort your comparison file, as Sara indicated, by the stratum in the PSU variables in the comparison weights file. It's comp STR and comp PSU. So we'll do that. Okay, now you're ready to run your comparison analysis. And so what I've done is I've set up some little examples of some output that you might be interested in looking at. The first one is simply a simple cross tab in SUDAAN that generates out a cross tab of the services variable by the NSCAW cohort in. Just note here, the cohort variable that's in the calibration weights file is called Ncohort. So that's where you see that. And it's run by the re-coded services variable. So we can just run that.
^M00:42:57
And so that'll generate chi-squared output. It yields a non-significant p-value of .6779. So they're not significantly different from each other. And then if you want to look at the distribution by cohort, you can see that the -- you can compare the row percents of a value of zero. So you've got 70.1 in cohort 1. Cohort 1 is this row. And then cohort 2, you can see that the percentage of zero values for service fee for cohort 2 is 71.79%. And so they're not significantly different. The next example is doing a similar cross tab, only this one I'm going to run the NSCAW cohort again by a caseworker risk assessment variable, one of those which indicates caseworker reporting active alcohol abuse by the primary care giver. And we'll see what this results in. And here you can see that there are some differences that you'll find that will be significant. And you can see the differences in the row percentages for NSCAW I and NSCAW II. You've got yeses that have -- they're 8.26% of the yeses for NSCAW I. And 4.88% of the yeses for NSCAW II. So that's just a little example of running a simple cross tab that you can see the differences between the two cohorts. Next we're going to run an example to see if there's a difference in the proportion receiving services in NSCAW I versus NSCAW II by using the re-coded services variable. So we'll run that.
^M00:45:23
And again, we can see from this output that there, again, is no significant difference between the means of that variable. And then this other sample code allows you to actually see what the two means are. And so you can see the mean in cohort 1 is .3 for that re-coded services variable. And it's .28 for NSCAW II versus, again, .3 for NSCAW I. Okay, and then finally, and this is something that is similar to the example that was discussed in the slides by Sara. Well, I also in my analysis file included the total score for the child's CBCL. So we can see what the differences look like for that when you compare NSCAW I to NSCAW II. And there's a non-significant difference with a p-value of .9619. So there isn't a significant difference between the two CBCL total scores. And then you can see what the means comparison looks like with this output. You can see that it's 53.14 in NSCAW I and the mean is 53.2 for NSCAW II. So you can see that there's a non-significant difference between the two. And then finally, just to show you that you can also run a -- you know, you can re-run regression models. This particular one is using the child's CBCL as the dependent variable. And the independent variables are the child's gender and the cohort. And again, you can see the output that's generated for that little regression model.
^M00:47:46
So that gives you some examples for making baseline comparisons between the two cohorts. Shortly you should be seeing some type of a message from NDACAN indicating that 18 month comparison weights will be available. And so, this next code that we'll be looking at makes the 18 month follow-up comparisons. And remember that with an 18 month follow-up comparison, it is, for NSCAW I, it's wave 3. And for NSCAW II, it's wave 2. So that's an important thing to remember. I don't think I'll take time by going through the steps that create the analysis file because they're basically the same as they were for the baseline comparisons except for a difference that you need to keep in mind. And that's re-naming. And what this example does to make it simple is it re-names the NSCAW I wave 3 variable, which is the child's CBCL, total score. It re-names it from a wave 3 naming convention, which is in the NSCAW I wave 3 file to the wave 2 naming convention. So, you need to do that because remember, you're stacking the two datasets. So when you stack, you have to make sure that your variable names are the same for those data files that you're stacking. So, what this example does is it simply re-names, like I said the NSCAW I wave 3 version of that variable from YB3 underscore TBT to YB2 underscore TBT. And then you can, once you have that and re-naming, then you can stack. And then again, the same thing, sorting by the comparison stratum and the comparison PSU variables. So that shows you some examples of making comparisons for the baseline and the 18 month comparisons.
^M00:50:25
So with that, Elliott, I'll throw it back to you.
>> [Elliott Smith:] Oh, great. Okay, thanks very much, Keith. Actually, while you were talking I got a couple of questions from Heather that I'll read out. And I think you addressed this first one, Keith, but I'll give you a chance to sort of answer again. Heather had asked: If you're using data from multiple waves, how should you make a decision about which calibration weights to use?
^E00:50:57
^B00:51:06
>> [Keith Smith:] So, is that -- Heather, does that mean derived variables?
>> [Elliott Smith:] Well, that's a good question. I think that she's alluding to -- well, I was hoping you could maybe point out to folks that just like your examples, when you're comparing NSCAW I and NSCAW II, you're doing that at each sort of cross section, right?
>> [Keith Smith:] Right. Yeah, at each, you know, follow-up comparison, so to speak, in other words within the baseline for the two cohorts and within the 18 month follow-up. And again, the same thing for once the 36 month follow-up weights come out, it would be the same thing. So it's making those comparisons -- you know, making the comparisons for those two follow-up waves for each cohort. If that makes any sense.
>> [Elliott Smith:] Yeah, I think so. I'm still getting used to this multitasking, because while I'm listening to you, I'm also reading what Heather is saying. So, Heather, if you want me to un-mute you, just in the chat box say yes.
>> [Keith Smith:] That might be best, I think. If Heather can ask the question.
>> [Elliott Smith:] Oh, yeah. That's fine. Okay, so let me bring her on. Okay, Heather, you should be able to. Can you hear us, Heather?
>> [Keith Smith:] Yeah, and if you're on mute, Heather, you should take off your mute.
^M00:53:20
>> [Elliott Smith:] Well, she should be allowed. Let's see. Sorry, guys. This is not working as smoothly as I hoped. So, I'm not able to -- I'm sorry. It's not working out. So I'll read her follow-up question. Heather had said: I was just thinking about the advice that we discovered earlier to use the wave that defines your population most recent data available. So if you're including wave 3 outcomes, use wave 3 weights?
^M00:54:05
>> [Paul Biemer:] Yeah. I mean, that --
>> [Keith Smith:] Yeah, Paul or Sara could --
>> [Paul Biemer:] Yeah, that's appropriate. I mean, you --
>> [Elliott Smith:] Can you maybe re-state it, Paul, so that everybody understands.
>> [Paul Biemer:] Okay, so let's suppose you, you know, you're making an estimate on NSCAW II for wave 3. We've developed or we will be developing calibration weights for that wave. You would use wave 3 weights to make that estimate. Now, when you go to NSCAW I, you know, you need to pick a wave. You know, probably the comparable wave in NSCAW I. You would use the weights associated with that wave. And maybe numbered differently than wave 3. What is the comparable wave in NSCAW I to wave 3 in NSCAW 2?
>> [Keith Smith:] Wave 4.
>> [Paul Biemer:] Wave 4. Okay, so you would use the wave 4 weight in NSCAW I, develop an estimate. You would wave 3 weight NSCAW II to develop and estimate. And then you would compare them. And as Sara said, you know, to be able to get the appropriate standard error, you would want to use a method that took into account the fact that these two samples are joined at the PSU level, which it is, it's a correlation. But that's not the question. The question is what weights you use.
>> [Elliott Smith:] Right. And so, Paul, if you were running analyses with like wave 3 outcomes, but you included in your regression model some variables from, say, baseline or wave 2, you would still go with, you think, the wave 3 weight at that time? Does that make sense?
>> [Paul Biemer:] Yeah. The wave 3 has been adjusted for the non-responsive wave 3. So, there's been some other adjustments, but I think that's the major one. So now if we're going to be using those children who responded at wave 3, then of course you would incur a non-response bias associated with those who did not respond at wave 3. The wave 3 calibration weight adjusts for that non-response bias. And if you were then going to bring in information from other waves for those children, the same non-response would apply. It would only be for the respondents at wave 3. So you want that wave 3 adjustment in the weight. So, you know, so what defines which weight you use is the population or the sample of respondents that you're using. Because when you go from wave 1, wave 2, wave 3 and so forth, essentially are, you know, you have different non-response. And these wave weights have been adjusted for the non-response at that particular wave.
^M00:57:23
>> [Elliott Smith:] So, that's very helpful. Yeah, and I know that that's been sort of -- and Heather mentioned sort of a point of confusion for some folks. But I think you made it really clear. Thanks. And also, too, another question from a little while ago. This is a very practical question. Do you think, or is there -- well, for folks that are writing articles that maybe are using the calibration weights, and in their journal manuscript submissions, you know, they need to explain the use of the calibration weights in their methods section. Is there any sort of a like journal-ready sort of text or samples or, you know, brief descriptions that folks might be able to refer to help, you know, responsibly explain the calibration weighting process? Does that make sense?
>> [Paul Biemer:] Yeah. Let me ask Keith if he can address that. We have developed internal documentation here at RTI. And I believe that internal documentation is currently under review to see if we can release it. Keith, do you know how close it is?
>> [Keith Smith:] Yeah. I mean, I know that we have that text. In fact, I'm not sure how widely distributed it is. But we can look into that. And I'm sure it's out there. But, I would need to know at what stage we're at in the review process so we can make that text available to users. So I'll talk to our internal analysis team to see if we can make that available for folks.
>> [Paul Biemer:] I would think that it would be part of the [inaudible] at some point, you know?
>> [Keith Smith:] Yeah, or at minimum the documentation for the calibration weights that we deliver.
>> [Elliott Smith:] Right. That would be a good place. So, I have a couple more questions coming in. And also I should let folks know that I'm clearly inept at operating all these controls. So don't try raising your hand. If you have a question for me, please send it in the chat box. So, I have a couple. Here's a question from Michelle. This gets to the distinction between intersection weights and calibration weights. Michelle says: Is it appropriate to pool samples from NSCAW I and II to build sample size, that is, combine the two datasets like we're talking about. If so, is the intersection approach or calibration-weighted approach most appropriate? So basically, you know, is there a time when intersection would be preferred over calibration? And also, does our statistical power, now, increase when we are putting both NSCAW samples or cohorts together?
^M01:00:42
>> [Paul Biemer:] So you -- so the question is, if I combine NSCAW I and NSCAW II, wave 1, for example, into one big file and I do an analysis that would basically be over both samples. How would I go about doing that? Well, the first question would be -- that I would ask is what population would that represent? Because you know, you have NSCAW I, which represents the child welfare population back in '99. And then you NSCAW II, which is, you know, nine years later. So, that would cause some problems in just -- I'm trying to understand what population that represents. Well, let's suppose that you were able to justify it somehow and just call it kind of a, you know, just some sort of an average population, which have been separated, you know, by nine years. But you just wanted to get some average for whatever that represents. Then, you know, you have two samples and each of them has their own weight. And so you can use the -- I mean, if you use the -- you know, I'm still struggling with the appropriateness of even combining these datasets. But I think maybe that -- one set of weights doesn't really offer any advantage over the other. Again, the calibration weights offers the advantage, when it's appropriate to use them, it offers the advantage of expanding inference from, you know, a 90% to 95% of the child welfare population. So you're getting a little bit more of the child welfare population represented in your estimates. And that's the reason we do that. It's a small difference, but it can mean a lot in terms of making comparisons between NSCAW I and NSCAW II. If you're not going to make comparisons between NSCAW I and NSCAW II, then I would say, you know, just use the NSCAW II weights, the standard weights. Because there are fewer complications associated with using them. They're easy to explain. They don't delete the 14 to 17 year olds. And you know, they're completely valid. It's just infers to a smaller, slightly, 5% smaller, population.
^M01:03:21
>> [Elliott Smith:] Okay, well, and then also, to follow up a little bit, with like a specific kind of analysis that I can imagine folks might be interested in doing. You know, I can imagine that folks would be interested in seeing if the relationship between, you know, a certain predictor and an outcome variable in NSCAW I, whether that relationship was similar for NSCAW II. So, I can image there'll be a lot of people that want to do sort of a cohort by predictor interaction to, you know, to see if there's sort of a time or cohort difference in those relationships. That's an appropriate use for these calibration weights, right?
>> [Paul Biemer:] Yes, right. I mean, that sounds like the example that Sara presented, where she was looking at CBCL scores.
>> [Elliott Smith:] Right. Right, exactly. But then actually, yeah, with the interaction term.
>> [Paul Biemer:] Yeah, you could do the interaction term in there. Yeah.
>> [Elliott Smith:] Okay. Okay, I think -- are there other questions that have come in?
^M01:04:54
The one -- oh, this may be -- this is a question from Megan. I apologize, Megan, if this too slow in bringing this up. But, I'll read it and then we can try to de-code it. So, Megan had asked: Is Paul saying that if we want to compare NSCAW I and II in analyses with baseline independent variables and 36 month outcomes, we use -- well, let's see. Yeah, we use a calibration weights, right, that apply -- I'm sort of editing her question -- calibration weights that apply for the 36 month follow-up. Yes. And that is exactly what we were just saying before.
>> [Paul Biemer:] So is the question if we want to compare differences within cohorts? Between cohorts?
>> [Elliott Smith:] Yeah, so say our predictors were all baseline. I think you've answered this already, Paul. All the independent variables are baseline, from the baseline. And then you have an outcome at 36 months. And then you had said, just to confirm that you had said earlier, that with that kind of analysis you would use the wave 3, or the 36 month weight because that would adjust for the non-response that you had.
>> [Paul Biemer:] Right. yeah, so that's right. So, yeah. Anyone who didn't respond to wave 3 would kind of fall out of that analysis.
>> [Elliott Smith:] Right.
>> [Paul Biemer:] And so then you would use wave 3 to adjust for that.
>> [Elliott Smith:] Okay. Cool. Now, do we have any other questions that folks would like to ask? I feel like some sort of strange translator. Let's see. [Inaudible]. Oh, yeah. So here at Cornell we received a question. So, just to let folks know, or just to remind people, so, the calibration weights will be available to restricted release users upon request. So you can email NDACAN support at Cornell.edu. If you don't already have some of the calibration weights, you can email us. We currently have at the archive the calibration weights for the baseline, so wave 1. And then soon we'll be providing the wave 2, right, the 18 month. And then down the road the 36 month. And then the timeline on that -- well, we'll send out an announcement when those 18 month calibration weights are available for folks.
>> [Keith Smith:] And just on that point, Elliott, those 18 month follow-up calibration weights should be available. We're hoping to send those to the archive within the next -- within the next week. So you know, it's going to be fairly soon when users can request those.
>> [Elliott Smith:] Okay. Thank you. That's great. And then also, too, in the documentation, in addition to the SUDAAN and SAS code, we have written some examples in FPSF for concatenating these sets together and that sort of thing. And I had a question from someone asking about a Stata version as well. And that's something that we'll be able to do here in the archive. And so, we'll --
>> [Keith Smith:] Well, actually, I think -- I'm sorry to interrupt, Elliott. We do have Stata examples as well in the documentation.
>> [Elliott Smith:] Oh, thank you. Okay. Okay, great. So, yeah. So the answer to that is, yeah, we'll have all the main, four main packages there. Okay. I haven't had any other questions come in. So, we can stop now. I want to thank Paul, Sara and Keith so much for giving the presentation today and helping us understand the calibration weights. It's an exciting opportunity to be able to compare national samples of the child welfare system, you know, eight or nine years apart. So that's a lot of great work that they've done. We will be posting the video from today on our Website so that folks that weren't here, or if you missed something and want to go back to it, you'll be able to. And also, for the folks on the all, I have a version of Keith's syntax, or the syntax that he went through today. And I'll be circulating that around. And I believe we'll also be able to circulate the slides form today, also. But of course they'll be on the video for folks, too. So, that's all we have here. Is there anything, Paul, Sara or Keith, that we need to say before we close off?
>> [Paul Biemer:] No, I don't think so. And it's been a real pleasure. I've enjoyed talking about this material.
>> [Elliott Smith:] Great. Well, thank you all very much. And have a great afternoon.
>> [Paul Biemer:] All right, you, too. Thank you very much, Elliott. Bye-bye.
>> [Elliott Smith:] Bye-bye.