Issues loading the latest version of the Canadian SNOMED CT Release dated 2022-03-31

Hi Gabor,
Thank you for taking the time to report your findings which I'm sure took a long time. There is certainly a lot to "unpack" in this message. We are currently assessing the potential sources of the issues you reported. We would like to take this opportunity to have a conversation with you, on these issues and the previous ones you have reported, so we are ensuring the highest quality of product for the Canadians stakeholders. Would you be able to send an inquiry to the Infoway helpdesk at Cette adresse courriel est protégée contre les robots spammeurs. Vous devez activer le JavaScript pour la visualiser. ?

Thank you!

Hi all,

I would also like to share our findings regarding the latest SNOMED CT Canada Edition release. While investigating Debbie's loading issue we found a few discrepancies in the package. Although they are not strictly related to the previously mentioned cases, I hope that sharing this information here will benefit everybody in the community.


Release package naming and Delta content

The latest release package was named as SnomedCT_Canadian_EditionRelease_PRODUCTION_20220331T120000Z.zip . This filename gives the impression that this package is an Edition release, meaning it contains both the International Edition and the Canada Extension content. It still sounds valid when reading the readme file within the archive:

The March 2022 CA Edition is a complete edition composed of the latest SNOMED CT International Edition and Canadian extensions. It
contains all concepts, descriptions, and relationships that are distributed by SNOMED International in monthly releases leading up
to March (January 2022 and February 2022 International Releases), as well as Canadian extension content including the English Canadian
(EN CA) content, French Canadian (FR CA) content, and Canadian reference set content.

The metadata file (release_package_information.json) in the package also suggests that the Delta contains everything that is necessary to perform an upgrade:
However, the Delta files do not contain any International SNOMED CT content since 2021-09-31. This fact renders the package to be a mix-up of an extension and an edition release, breaking the contract the official SNOMED CT Release Format specification offers.

According to the Release Package Naming Convention :

Using the word "Edition" means:
The release files included in the package fully resolve all dependencies of all modules included in the package.

The individual filenames also contain the word "Edition", see:

I see the good intentions with keeping the Delta files within the release package, but this way it could be dangerous to those who got used to consuming Delta files. Since the introduction of Frequent Delivery releases, all the toolings had to prepare for release packages without Delta files. This means that - ideally - the Full files should be sufficient for any consumers going forward. A good example of this approach is the SNOMED CT US Edition, where the latest release package follows the same format as SNOMED International, no Delta is included.

I have to add though, that using the Full files is also somewhat tricky in this case, which I'm describing below why.


Unfortunate timing of content donation

147 concepts became the first "victims" of the International Frequent Delivery releases. These concepts were all introduced first in the Canada Extension with the effective time 2021-09-30. Then they were donated to the International Edition, where they got accepted and released with an effective time of 2021-09-30.

Let me show you through an example why this situation is a bit worrying.

30421000087105 - Corynebacterium mustelae (organism)

All corresponding RF2 rows in sct2_Concept_Full_CanadianEdition_20210930.txt:

All corresponding RF2 rows in sct2_Concept_Full_CanadianEdition_20220331.txt:

The historical information of these concepts has disappeared between the 2021-09-30 and the 2022-03-31 Canada Edition releases.

The modules of these concepts are not represented properly within Snowstorm either. See the modules ...

... in the International Edition:
browser.ihtsdotools.org/?perspective=full&conceptId1=30421000087105&edition=MAIN/2022-04-30&release=&languages=en

... in the Canada Edition:
browser.ihtsdotools.org/?perspective=full&conceptId1=30421000087105&edition=MAIN/SNOMEDCT-CA/2022-03-31&release=&languages=en,fr

The official RF2 Delta Generator Tool cannot be used here because of the clashing effective times.

On the other hand, the RF2 specification says:

Once a record has been released in a release file, it will continue to be released in exactly the same form in future release files.

Using the module dependency reference set could be a potential solution, but that alone does not fix the issues within the RF2 files. The latest Canadian Full files do contain the correct International version of those concepts, yet it breaks the RF2 standard itself.


Missing components between releases

Last but not least, a less critical issue has also been identified. As the specification says, rows from the RF2 release files should not disappear ever, yet there is content that was part of the 2021-09-30 Full release and is not present in the 2022-03-31 Full release files.

We have found:
- 4468 Simple Map reference set members
- 1 module dependency reference set member
that is missing from the latest release package, but was present in the previous one.


We will help Debbie to import the latest Canadian release into their authoring environment, but in case there is a possibility for a patch release, please let us know.

Regards,

Gábor
B2i Healthcare

Not sure I follow you. A namespace for snowstorm? you mean a DNS?

Thank you for the update Anibal!
Do you have a namespace?

Just want to provide an update with respect to this and what ended up happening on our side.

So, what I really wanted to do was to upgrade our instance of snowstorm with the latest version of the Canadian release. I was able to do it, but we were getting really bad performance on the server, with the queries using the snowstorm API taking about 6-8 times longer than expected.

After checking we noticed that elasticsearch was having issues and we tried to fix it with Kai but couldn't so he suggested to start from scratch and only load the latest version. So, that's what I did and then I had the issues that lead to this post.

Right now the server is operating wonderfully, but I was not able to just load the Canadian release, as Rory had suggested originally (since he told me that the Canadian release is bundled with the International release). Snowstorm was loaded with the Canadian release but there was nothing been returned by my queries. So, he suggested that I load the International release first and he gave me access to the February International Release. I loaded that one, then the Canadian March release and voila.

Hi Debbie,

Most implementations would use the Canadian Edition release files, that resolve all the dependencies (Canadian extensions, SI International Edition January 2022+February 2022 monthly update) in a single package. Loading the Canadian Edition on Snowstorm (with all dependencies already resolved) works fine.

However, extension developers extending the Canadian extensions would use the March Canadian extension distribution to update their development environments rather than the precomputed Canadian Edition release. Since the March Canadian extension (and edition) depend on the February SI release, I would consider loading first the SI February release to update the SI January release, and only after satisfying that dependency load the Canadian extension. Otherwise, there might be missing references (integrity errors) as the Canadian content might reference February content that was not present in the January SI release, causing unexpected loading errors.

Snowstorm can be used in both ways, as Anibal pointed out: loading individual extensions (taking care of satisfying the required dependencies) or loading the precomputed [Canadian] Edition. We see better performance in read-only production environments using the later option as version computing chores are dramatically decreased.

Debbie, I understand that as an extension developer extending the Canadian release you have to use the extension package. If your loading and integrity testing fails, I would try first to update your SI International Edition content to the February release, and only then load the March Canadian extension update.

SNOMED CT International Edition monthly releases are a significant step forward, giving implementations and extension developers the opportunity to access current content and decrease the version offset between localized and international versions. However, those using the extension (rather than the edition) would need to manually check that the dependencies are satisfied, at least until tools start to read the module dependency reference set and fail when those dependencies are not satisfied.

Note that users that are not extending the Canadian extensions in a SNOMED CT authoring environment could just use the Canadian Edition distribution and avoid all the potential issues represented by dependencies and conflicts between them. As Kai said, the potential for integrity issues during content import will likely be detected in future versions of the tools.

Hope this helps, I apologize in advance if I have misunderstood the issue. I have not been able to test the issue and possible workarounds.

Cordially
Guillermo Reynoso
termMed Terminology Team