The GDIT and the Call Center CSV files have very different schemas. I haven't looked into it in much detail yet, but I don't think they are even supposed to map perfectly.
This also explains why there is so much difference between them in terms of the age field.
For starters, there are 690722 registrants with AGE_AT_VAX in Call Center CSV file, and 791891 registrants with AGE_AT_VAX in the GDIT CSV file. Even if the GDIT file size may be smaller, clearly it has info on the age of a lot more people.
But neither is a superset of the other. There are 104659 registrants in GDIT CSV with age, but don't have age in the Call Center CSV file. There are also 3490 registrants in Call Center CSV with age, which don't have age in the GDIT CSV file.
Someone has to investigate these disparities, for now I am more focused on the free text so you can take a look if you are interested.
The ages will be the cream, but not my main focus right now right either it just came to light for me trying to figure out how ICAN arrived at 2.77M unique registrants with ages. It's clear they did not make any accommodation for ID#s with a survey static ID# ending in "-Dep" aka dependent. I guess along with all the deaths, dependents were the two main objectives for me of getting these ID#s included in your ~800K records. I knew the the two call center files were different structures and would impact the file size in a two dimensional sense, but as I began matching ID#'s from both files I can see exactly what you describe as "one not being a superset of the other. I'll get it it sorted out from my end. I guess the important part that we can reconcile the final master list of ID#'s that will be available in your report tool. I know many of the dependents did not qualify in your parameters as there were many kids with seemingly nothing wrong and just participated in a survey as a matter of prudence. That's ok, just calling out attention that there is at least 172K kids in the system is a huge step forward.
The GDIT and the Call Center CSV files have very different schemas. I haven't looked into it in much detail yet, but I don't think they are even supposed to map perfectly.
This also explains why there is so much difference between them in terms of the age field.
For starters, there are 690722 registrants with AGE_AT_VAX in Call Center CSV file, and 791891 registrants with AGE_AT_VAX in the GDIT CSV file. Even if the GDIT file size may be smaller, clearly it has info on the age of a lot more people.
But neither is a superset of the other. There are 104659 registrants in GDIT CSV with age, but don't have age in the Call Center CSV file. There are also 3490 registrants in Call Center CSV with age, which don't have age in the GDIT CSV file.
Someone has to investigate these disparities, for now I am more focused on the free text so you can take a look if you are interested.
The ages will be the cream, but not my main focus right now right either it just came to light for me trying to figure out how ICAN arrived at 2.77M unique registrants with ages. It's clear they did not make any accommodation for ID#s with a survey static ID# ending in "-Dep" aka dependent. I guess along with all the deaths, dependents were the two main objectives for me of getting these ID#s included in your ~800K records. I knew the the two call center files were different structures and would impact the file size in a two dimensional sense, but as I began matching ID#'s from both files I can see exactly what you describe as "one not being a superset of the other. I'll get it it sorted out from my end. I guess the important part that we can reconcile the final master list of ID#'s that will be available in your report tool. I know many of the dependents did not qualify in your parameters as there were many kids with seemingly nothing wrong and just participated in a survey as a matter of prudence. That's ok, just calling out attention that there is at least 172K kids in the system is a huge step forward.