From bgilbert+openslide at cs.cmu.edu  Sun May  5 12:16:22 2024
From: bgilbert+openslide at cs.cmu.edu (Benjamin Gilbert)
Date: Sun, 5 May 2024 11:16:22 -0500
Subject: OpenSlide Java 0.13.0 released
Message-ID: <CANioFng8nfWUxQXDfCGe8J89zAJCmC-eaDHMrcWF5LZxhSLqeQ@mail.gmail.com>

I'm pleased to announce the release of OpenSlide Java 0.13.0.
OpenSlide Java is the official Java-language binding to OpenSlide, but
does not yet have a stable API.

Starting with this release, OpenSlide Java loads OpenSlide via Java's
new Foreign Function and Memory API instead of the legacy Java Native
Interface.  This eliminates the need for a platform-specific glue
library, allowing OpenSlide Java to be shipped as a single
platform-independent JAR.  With this change, OpenSlide Java uses any
copy of OpenSlide already loaded into the JVM with System.load() or
System.loadLibrary(), and otherwise loads OpenSlide from the operating
system's library search path.

This release also adds support for OpenSlide 4.0.0's cache management
and ICC color profile APIs.  In addition, it removes the Autotools+Ant
and Meson build systems in favor of Maven.

OpenSlide Java 0.13.0 requires Java ? 22 and OpenSlide ? 4.0.0.  Users
with older versions of these dependencies should continue using
OpenSlide Java 0.12.4 until they can upgrade.

Alongside this release, OpenSlide binary build 4.0.0.3 removes
OpenSlide Java, since platform-specific Java builds are no longer
necessary.  A prebuilt JAR is available as part of the OpenSlide Java
release.

The new releases are available at:

    https://openslide.org/download/

The full changelog is below.

Thanks,
--Benjamin Gilbert


## Breaking changes

* Require Java ? 22
* Require OpenSlide ? 4.0.0
* Switch from JNI to FFM, making OpenSlide Java builds platform-independent
* Build with Maven
* Remove Autotools+Ant and Meson build systems
* Stop shipping source tarball, in favor of GitHub source archive
* Rename JAR to openslide-java-$version.jar

## New features

* Support OpenSlide cache management API
* Support reading ICC color profiles from slides and associated images
* Add OpenSlide.readRegion() method returning a BufferedImage
* Add OpenSlide.PROPERTY_NAME_ICC_SIZE
* Automatically close unreferenced OpenSlide and OpenSlideCache objects
* Add one-argument OpenSlideDisposedException constructor
* Ship JAR as release artifact

## Deprecations

* Deprecate OpenSlide.dispose() in favor of OpenSlide.close()
* Deprecate zero-argument OpenSlideDisposedException constructor

From 2693926255 at qq.com  Mon May  6 06:35:07 2024
From: 2693926255 at qq.com (=?gb18030?B?0Oy/rA==?=)
Date: Mon, 6 May 2024 18:35:07 +0800
Subject: Supports direct operation of.kfb files?
Message-ID: <tencent_D504A340E168DA93484EAFE3D9DC279EF405@qq.com>

Hello! I am a Chinese user. May I ask whether the products of your company (openslide) can support the direct operation of the.KFB format file, which is from a medical enterprise in Zhejiang Province, China? Thank you so much!!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/openslide-users/attachments/20240506/5a3bfa64/attachment.html>

From bgilbert+openslide at cs.cmu.edu  Mon May  6 13:44:55 2024
From: bgilbert+openslide at cs.cmu.edu (Benjamin Gilbert)
Date: Mon, 6 May 2024 12:44:55 -0500
Subject: Supports direct operation of.kfb files?
In-Reply-To: <tencent_D504A340E168DA93484EAFE3D9DC279EF405@qq.com>
References: <tencent_D504A340E168DA93484EAFE3D9DC279EF405@qq.com>
Message-ID: <CANioFngRXjt7ch9hj_OpAMnrPbQJ+MohnriDYZzHi2YrGZ=SOg@mail.gmail.com>

Hello,

On Mon, May 6, 2024 at 8:09?AM ?? via openslide-users
<openslide-users at lists.andrew.cmu.edu> wrote:
> May I ask whether the products of your company (openslide) can support the direct operation of the.KFB format file, which is from a medical enterprise in Zhejiang Province, China?

OpenSlide currently does not support KFB, but contributions of code
and sample data are welcome.  Further discussion here:
https://github.com/openslide/openslide/issues/592

Best,
--Benjamin Gilbert

From 2693926255 at qq.com  Mon May  6 20:36:28 2024
From: 2693926255 at qq.com (=?gb18030?B?0Oy/rA==?=)
Date: Tue, 7 May 2024 08:36:28 +0800
Subject: =?gb18030?B?u9i4tKO6IFN1cHBvcnRzIGRpcmVjdCBvcGVyYXRp?=
 =?gb18030?B?b24gb2Yua2ZiIGZpbGVzPw==?=
In-Reply-To: <CANioFngRXjt7ch9hj_OpAMnrPbQJ+MohnriDYZzHi2YrGZ=SOg@mail.gmail.com>
References: <tencent_D504A340E168DA93484EAFE3D9DC279EF405@qq.com>
 <CANioFngRXjt7ch9hj_OpAMnrPbQJ+MohnriDYZzHi2YrGZ=SOg@mail.gmail.com>
Message-ID: <tencent_7DE0F59A9EFB7C82486F304E142EA75B8809@qq.com>

Thank you for your reply! Currently, due to medical privacy concerns, we are unable to provide the .kfb format file. However, we will upload sample files that are permitted to be made public in the future. Thank you again, and we hope OpenSlide becomes even more powerful!




------------------&nbsp;????&nbsp;------------------
???:                                                                                                                        "openslide-users"                                                                                    <bgilbert+openslide at cs.cmu.edu&gt;;
????:&nbsp;2024?5?7?(???) ??1:44
???:&nbsp;"openslide-users"<openslide-users at lists.andrew.cmu.edu&gt;;
??:&nbsp;"??"<2693926255 at qq.com&gt;;
??:&nbsp;Re: Supports direct operation of.kfb files?



Hello,

On Mon, May 6, 2024 at 8:09?AM ?? via openslide-users
<openslide-users at lists.andrew.cmu.edu&gt; wrote:
&gt; May I ask whether the products of your company (openslide) can support the direct operation of the.KFB format file, which is from a medical enterprise in Zhejiang Province, China?

OpenSlide currently does not support KFB, but contributions of code
and sample data are welcome.&nbsp; Further discussion here:
https://github.com/openslide/openslide/issues/592

Best,
--Benjamin Gilbert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/openslide-users/attachments/20240507/413244dd/attachment.html>

From m.weihrauch at smartinmedia.com  Sat May 11 15:15:49 2024
From: m.weihrauch at smartinmedia.com (Martin Weihrauch)
Date: Sat, 11 May 2024 19:15:49 +0000
Subject: Question about NDPI
In-Reply-To: <CANioFng8nfWUxQXDfCGe8J89zAJCmC-eaDHMrcWF5LZxhSLqeQ@mail.gmail.com>
References: <CANioFng8nfWUxQXDfCGe8J89zAJCmC-eaDHMrcWF5LZxhSLqeQ@mail.gmail.com>
Message-ID: <FR0P281MB2649DAEB3BA43CE232864E9892E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>

Dear community,

my C-knowledge does not suffice to read the entire code of openslide-vendor-hamamatsu.c, but I am interested to better understand the Hamamatsu format.

If I see it correctly, it (ab)uses the Tiff format and stores a single JPEG in each zoom level, (ab)using the JPEG rules, e. g. by exceeding the 65,000 pixel limit, etc.

My question: how is it possible to quickly extract a tile from the large JPEG? Does it internally have multiple frames (the stripes) and if yes, how is it possible to locate the internal MCU/8x8 blocks without reading the entire stripe or does each stripe have to be "parsed" at least once completely? Is there something like a directory involved?

Thank you 

Martin




From bgilbert+openslide at cs.cmu.edu  Sat May 11 16:06:37 2024
From: bgilbert+openslide at cs.cmu.edu (Benjamin Gilbert)
Date: Sat, 11 May 2024 15:06:37 -0500
Subject: Question about NDPI
In-Reply-To: <FR0P281MB2649DAEB3BA43CE232864E9892E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
References: <CANioFng8nfWUxQXDfCGe8J89zAJCmC-eaDHMrcWF5LZxhSLqeQ@mail.gmail.com>
 <FR0P281MB2649DAEB3BA43CE232864E9892E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
Message-ID: <CANioFnj1tGMZn0XySCyinVoO9CjOu9GyaBnCceZ-YME+91sv0g@mail.gmail.com>

On Sat, May 11, 2024 at 2:15?PM Martin Weihrauch
<m.weihrauch at smartinmedia.com> wrote:
> If I see it correctly, it (ab)uses the Tiff format and stores a single JPEG in each zoom level, (ab)using the JPEG rules, e. g. by exceeding the 65,000 pixel limit, etc.

Right.

> My question: how is it possible to quickly extract a tile from the large JPEG? Does it internally have multiple frames (the stripes) and if yes, how is it possible to locate the internal MCU/8x8 blocks without reading the entire stripe or does each stripe have to be "parsed" at least once completely? Is there something like a directory involved?

An NDPI tile is actually a sequence of MCUs between two JPEG restart
markers.  Restart markers are a JPEG feature that isn't normally used
much; they allow the decoder to recover from data corruption.  Restart
markers can be searched for without decoding the image data, and they
reset the state of the encoder/decoder when encountered, so it's
possible to start decoding at any restart marker without knowledge of
previous MCUs.  OpenSlide reads a tile by concatenating the JPEG
header with the tile's MCUs, fixing up the trailer marker and the
header width/height fields as necessary, and passing the result to the
JPEG decoder.

Restart markers are placed after every N MCUs, and NDPI TIFF tag 65426
lists the byte offset (within the JPEG) of each MCU that immediately
follows a restart marker.  OpenSlide can also scan for restart markers
if that tag is missing.

Best,
--Benjamin Gilbert

From cgohlke at cgohlke.com  Sat May 11 16:20:48 2024
From: cgohlke at cgohlke.com (Christoph Gohlke)
Date: Sat, 11 May 2024 13:20:48 -0700
Subject: Question about NDPI
In-Reply-To: <FR0P281MB2649DAEB3BA43CE232864E9892E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
References: <CANioFng8nfWUxQXDfCGe8J89zAJCmC-eaDHMrcWF5LZxhSLqeQ@mail.gmail.com>
 <FR0P281MB2649DAEB3BA43CE232864E9892E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
Message-ID: <778c3902-66c2-448c-aa8c-1b75ee9e23d1@cgohlke.com>



On 5/11/2024 12:15 PM, Martin Weihrauch wrote:
> Dear community,
> 
> my C-knowledge does not suffice to read the entire code of openslide-vendor-hamamatsu.c, but I am interested to better understand the Hamamatsu format.
> 
> If I see it correctly, it (ab)uses the Tiff format and stores a single JPEG in each zoom level, (ab)using the JPEG rules, e. g. by exceeding the 65,000 pixel limit, etc.
> 
> My question: how is it possible to quickly extract a tile from the large JPEG? Does it internally have multiple frames (the stripes) and if yes, how is it possible to locate the internal MCU/8x8 blocks without reading the entire stripe or does each stripe have to be "parsed" at least once completely? Is there something like a directory involved?
> 

Yes, the TIFF tags 65426 and 65432 (if >4 GB) should contain the lower 
and higher bytes of the offsets to the MCU blocks relative to the strip 
offset in tag #273 (StripOffsets). The JPEG header at StripOffsets[0] 
with length McuOffsets[0] can be prepended to a MCU block to decode just 
the image in the block.

Here's some Python code to get the MCU offsets: 
<https://github.com/cgohlke/tifffile/blob/9000e7463df6f0a7dfdc72c701796967fbffd4df/tifffile/tifffile.py#L8489-L8500>

Hope this helps,

Christoph

From m.weihrauch at smartinmedia.com  Sat May 11 16:41:43 2024
From: m.weihrauch at smartinmedia.com (Martin Weihrauch)
Date: Sat, 11 May 2024 20:41:43 +0000
Subject: Question about NDPI
In-Reply-To: <CANioFnj1tGMZn0XySCyinVoO9CjOu9GyaBnCceZ-YME+91sv0g@mail.gmail.com>
References: <CANioFng8nfWUxQXDfCGe8J89zAJCmC-eaDHMrcWF5LZxhSLqeQ@mail.gmail.com>
 <FR0P281MB2649DAEB3BA43CE232864E9892E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnj1tGMZn0XySCyinVoO9CjOu9GyaBnCceZ-YME+91sv0g@mail.gmail.com>
Message-ID: <FR0P281MB264930E2262DE9748BFAEC9F92E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>

Thank you, Benjamin and Christoph, that explains it completely and very well! A very interesting approach to storing tiles.

Best

Martin


-----Original Message-----
From: Benjamin Gilbert <bgilbert+openslide at cs.cmu.edu> 
Sent: Samstag, 11. Mai 2024 22:07
To: openslide-users at lists.andrew.cmu.edu
Cc: Martin Weihrauch <m.weihrauch at smartinmedia.com>
Subject: Re: Question about NDPI

[Sie erhalten nicht h?ufig E-Mails von bgilbert+openslide at cs.cmu.edu. Weitere Informationen, warum dies wichtig ist, finden Sie unter https://aka.ms/LearnAboutSenderIdentification ]

On Sat, May 11, 2024 at 2:15?PM Martin Weihrauch <m.weihrauch at smartinmedia.com> wrote:
> If I see it correctly, it (ab)uses the Tiff format and stores a single JPEG in each zoom level, (ab)using the JPEG rules, e. g. by exceeding the 65,000 pixel limit, etc.

Right.

> My question: how is it possible to quickly extract a tile from the large JPEG? Does it internally have multiple frames (the stripes) and if yes, how is it possible to locate the internal MCU/8x8 blocks without reading the entire stripe or does each stripe have to be "parsed" at least once completely? Is there something like a directory involved?

An NDPI tile is actually a sequence of MCUs between two JPEG restart markers.  Restart markers are a JPEG feature that isn't normally used much; they allow the decoder to recover from data corruption.  Restart markers can be searched for without decoding the image data, and they reset the state of the encoder/decoder when encountered, so it's possible to start decoding at any restart marker without knowledge of previous MCUs.  OpenSlide reads a tile by concatenating the JPEG header with the tile's MCUs, fixing up the trailer marker and the header width/height fields as necessary, and passing the result to the JPEG decoder.

Restart markers are placed after every N MCUs, and NDPI TIFF tag 65426 lists the byte offset (within the JPEG) of each MCU that immediately follows a restart marker.  OpenSlide can also scan for restart markers if that tag is missing.

Best,
--Benjamin Gilbert

From m.weihrauch at smartinmedia.com  Mon May 13 23:30:06 2024
From: m.weihrauch at smartinmedia.com (Martin Weihrauch)
Date: Tue, 14 May 2024 03:30:06 +0000
Subject: Question about NDPI
In-Reply-To: <CANioFnj1tGMZn0XySCyinVoO9CjOu9GyaBnCceZ-YME+91sv0g@mail.gmail.com>
References: <CANioFng8nfWUxQXDfCGe8J89zAJCmC-eaDHMrcWF5LZxhSLqeQ@mail.gmail.com>
 <FR0P281MB2649DAEB3BA43CE232864E9892E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnj1tGMZn0XySCyinVoO9CjOu9GyaBnCceZ-YME+91sv0g@mail.gmail.com>
Message-ID: <FR0P281MB2649050C26EA90894147192B92E32@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>

I have another question regarding Hamamatsu NDPI's Tiff structure:

OpenSlide way (as found in openslide-decode-tifflike.c):
After carefully reading your code and the documentation, I understand that Hamamatsu chose to use the standard Tiff (non-BigTiff) and after the very first 2 bytes of the file (bytes #0 and #1, endianness) and the byte #2 for standard Tiff (2A) or BigTiff (2B), they write 8 bytes (against the standard Tiff rule with only 4 byte offset) for the offset to their first IFD/Tiff Directory.

In OpenSlide, you try guessing if it is NDPI by trying to read the first IFD from that theoretical 8 byte and check, if you find the tag 64520, which is "pathognomonic" only for NDPI. Else, you "fall back" to a standard Tiff file.

My findings:
At Smart In Media, have a huge slide collection and our earliest ever NDPI slide dates back to Oct 2007 and the latest to this year. The models were S210, S360 and S20 and one from one, where I am not sure (2007).
In all of them, I found the word "Hamamatsu" in a Hex Editor starting at byte #12 (in the oldest slide from 2007, it starts at #18). This would make it a lot easier at identifying Hamamatsu NDPI files from the content of the file, irrespective of the extension, by just searching the first e. g. 30 bytes for the string "Hamamatsu". 

Can you confirm this or do you have other findings?

Thank you 

Martin


 

-----Original Message-----
From: Benjamin Gilbert <bgilbert+openslide at cs.cmu.edu> 
Sent: Samstag, 11. Mai 2024 22:07
To: openslide-users at lists.andrew.cmu.edu
Cc: Martin Weihrauch <m.weihrauch at smartinmedia.com>
Subject: Re: Question about NDPI

[Sie erhalten nicht h?ufig E-Mails von bgilbert+openslide at cs.cmu.edu. Weitere Informationen, warum dies wichtig ist, finden Sie unter https://aka.ms/LearnAboutSenderIdentification ]

On Sat, May 11, 2024 at 2:15?PM Martin Weihrauch <m.weihrauch at smartinmedia.com> wrote:
> If I see it correctly, it (ab)uses the Tiff format and stores a single JPEG in each zoom level, (ab)using the JPEG rules, e. g. by exceeding the 65,000 pixel limit, etc.

Right.

> My question: how is it possible to quickly extract a tile from the large JPEG? Does it internally have multiple frames (the stripes) and if yes, how is it possible to locate the internal MCU/8x8 blocks without reading the entire stripe or does each stripe have to be "parsed" at least once completely? Is there something like a directory involved?

An NDPI tile is actually a sequence of MCUs between two JPEG restart markers.  Restart markers are a JPEG feature that isn't normally used much; they allow the decoder to recover from data corruption.  Restart markers can be searched for without decoding the image data, and they reset the state of the encoder/decoder when encountered, so it's possible to start decoding at any restart marker without knowledge of previous MCUs.  OpenSlide reads a tile by concatenating the JPEG header with the tile's MCUs, fixing up the trailer marker and the header width/height fields as necessary, and passing the result to the JPEG decoder.

Restart markers are placed after every N MCUs, and NDPI TIFF tag 65426 lists the byte offset (within the JPEG) of each MCU that immediately follows a restart marker.  OpenSlide can also scan for restart markers if that tag is missing.

Best,
--Benjamin Gilbert

From bgilbert+openslide at cs.cmu.edu  Tue May 14 00:00:07 2024
From: bgilbert+openslide at cs.cmu.edu (Benjamin Gilbert)
Date: Mon, 13 May 2024 23:00:07 -0500
Subject: Question about NDPI
In-Reply-To: <FR0P281MB2649050C26EA90894147192B92E32@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
References: <CANioFng8nfWUxQXDfCGe8J89zAJCmC-eaDHMrcWF5LZxhSLqeQ@mail.gmail.com>
 <FR0P281MB2649DAEB3BA43CE232864E9892E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnj1tGMZn0XySCyinVoO9CjOu9GyaBnCceZ-YME+91sv0g@mail.gmail.com>
 <FR0P281MB2649050C26EA90894147192B92E32@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
Message-ID: <CANioFniVz_dN5x-K=Mq=DUp80VR8LGa9c3x5cT_6Mf=CzchFhw@mail.gmail.com>

On Mon, May 13, 2024 at 10:30?PM Martin Weihrauch
<m.weihrauch at smartinmedia.com> wrote:
> At Smart In Media, have a huge slide collection and our earliest ever NDPI slide dates back to Oct 2007 and the latest to this year. The models were S210, S360 and S20 and one from one, where I am not sure (2007).
> In all of them, I found the word "Hamamatsu" in a Hex Editor starting at byte #12 (in the oldest slide from 2007, it starts at #18). This would make it a lot easier at identifying Hamamatsu NDPI files from the content of the file, irrespective of the extension, by just searching the first e. g. 30 bytes for the string "Hamamatsu".
>
> Can you confirm this or do you have other findings?

That approach is probably okay, though personally I wouldn't rely on
it.  That string "Hamamatsu" is referenced by the Make tag (271) of
the first TIFF directory, and so its location within the file is
probably not contractual.  For example, any other software that
decided to generate NDPI-compatible files would not necessarily put
the string in the same place, if it included the string at all.

Best,
--Benjamin Gilbert

From m.weihrauch at smartinmedia.com  Fri May 17 11:00:10 2024
From: m.weihrauch at smartinmedia.com (Martin Weihrauch)
Date: Fri, 17 May 2024 15:00:10 +0000
Subject: AW: Question about NDPI
In-Reply-To: <CANioFniVz_dN5x-K=Mq=DUp80VR8LGa9c3x5cT_6Mf=CzchFhw@mail.gmail.com>
References: <CANioFng8nfWUxQXDfCGe8J89zAJCmC-eaDHMrcWF5LZxhSLqeQ@mail.gmail.com>
 <FR0P281MB2649DAEB3BA43CE232864E9892E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnj1tGMZn0XySCyinVoO9CjOu9GyaBnCceZ-YME+91sv0g@mail.gmail.com>
 <FR0P281MB2649050C26EA90894147192B92E32@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFniVz_dN5x-K=Mq=DUp80VR8LGa9c3x5cT_6Mf=CzchFhw@mail.gmail.com>
Message-ID: <FR0P281MB264974ECDCEB62EF7985DD9E92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>

Thank you again for the great explanation!

Another issue for NDPI:
As the value/offset tags within the IFDs are only 4 bytes in Hamamatsu, I saw from OpenSlide that
- If it is not the 1st IFD, then it compares the offset of any tag with the offset to the tag from the 1st IFD. If the offset is the same --> take this 32bit offset, else combine the high bits of the 64bit offset of the IFD in the file with the tag 32 bit offset to 64bits. 

The question:
Isn't there the theoretical chance that the 32bit and the 64 bit offset could be the same though they are not?

E. g., let's assume, the label image is at the real offset of byte 17,000,000,123.
There is also another image at an offset of 123.
Isn't it possible that in the IFD, 123 is stored in the 32 bit and by looking at the first IFD, which also has an offset of 123, the software incorrectly assumes that this is the true offset instead of the one in the 17GB?

Best

Martin


-----Urspr?ngliche Nachricht-----
Von: Benjamin Gilbert <bgilbert+openslide at cs.cmu.edu> 
Gesendet: Dienstag, 14. Mai 2024 06:00
An: openslide-users at lists.andrew.cmu.edu
Cc: Martin Weihrauch <m.weihrauch at smartinmedia.com>
Betreff: Re: Question about NDPI

[Sie erhalten nicht h?ufig E-Mails von bgilbert+openslide at cs.cmu.edu. Weitere Informationen, warum dies wichtig ist, finden Sie unter https://aka.ms/LearnAboutSenderIdentification ]

On Mon, May 13, 2024 at 10:30?PM Martin Weihrauch <m.weihrauch at smartinmedia.com> wrote:
> At Smart In Media, have a huge slide collection and our earliest ever NDPI slide dates back to Oct 2007 and the latest to this year. The models were S210, S360 and S20 and one from one, where I am not sure (2007).
> In all of them, I found the word "Hamamatsu" in a Hex Editor starting at byte #12 (in the oldest slide from 2007, it starts at #18). This would make it a lot easier at identifying Hamamatsu NDPI files from the content of the file, irrespective of the extension, by just searching the first e. g. 30 bytes for the string "Hamamatsu".
>
> Can you confirm this or do you have other findings?

That approach is probably okay, though personally I wouldn't rely on it.  That string "Hamamatsu" is referenced by the Make tag (271) of the first TIFF directory, and so its location within the file is probably not contractual.  For example, any other software that decided to generate NDPI-compatible files would not necessarily put the string in the same place, if it included the string at all.

Best,
--Benjamin Gilbert

From bgilbert+openslide at cs.cmu.edu  Fri May 17 13:30:25 2024
From: bgilbert+openslide at cs.cmu.edu (Benjamin Gilbert)
Date: Fri, 17 May 2024 12:30:25 -0500
Subject: Question about NDPI
In-Reply-To: <FR0P281MB264974ECDCEB62EF7985DD9E92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
References: <CANioFng8nfWUxQXDfCGe8J89zAJCmC-eaDHMrcWF5LZxhSLqeQ@mail.gmail.com>
 <FR0P281MB2649DAEB3BA43CE232864E9892E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnj1tGMZn0XySCyinVoO9CjOu9GyaBnCceZ-YME+91sv0g@mail.gmail.com>
 <FR0P281MB2649050C26EA90894147192B92E32@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFniVz_dN5x-K=Mq=DUp80VR8LGa9c3x5cT_6Mf=CzchFhw@mail.gmail.com>
 <FR0P281MB264974ECDCEB62EF7985DD9E92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
Message-ID: <CANioFnjdXYqh-C11tifsT2RjXV2xUcofE5bpZ7qV17QzsBWaqw@mail.gmail.com>

On Fri, May 17, 2024 at 10:01?AM Martin Weihrauch
<m.weihrauch at smartinmedia.com> wrote:
> As the value/offset tags within the IFDs are only 4 bytes in Hamamatsu, I saw from OpenSlide that
> - If it is not the 1st IFD, then it compares the offset of any tag with the offset to the tag from the 1st IFD. If the offset is the same --> take this 32bit offset, else combine the high bits of the 64bit offset of the IFD in the file with the tag 32 bit offset to 64bits.

It turns out that this heuristic is incorrect.  Nicholas Trahearn
found that the high bits are actually present elsewhere in the file.
I haven't dug into this yet, but you can see his work at
<https://github.com/openslide/openslide/pull/602>.  I'm hoping to
switch OpenSlide to the new approach soon.

Best,
--Benjamin Gilbert

From m.weihrauch at smartinmedia.com  Fri May 17 14:26:27 2024
From: m.weihrauch at smartinmedia.com (Martin Weihrauch)
Date: Fri, 17 May 2024 18:26:27 +0000
Subject: Question about NDPI
In-Reply-To: <CANioFnjdXYqh-C11tifsT2RjXV2xUcofE5bpZ7qV17QzsBWaqw@mail.gmail.com>
References: <CANioFng8nfWUxQXDfCGe8J89zAJCmC-eaDHMrcWF5LZxhSLqeQ@mail.gmail.com>
 <FR0P281MB2649DAEB3BA43CE232864E9892E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnj1tGMZn0XySCyinVoO9CjOu9GyaBnCceZ-YME+91sv0g@mail.gmail.com>
 <FR0P281MB2649050C26EA90894147192B92E32@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFniVz_dN5x-K=Mq=DUp80VR8LGa9c3x5cT_6Mf=CzchFhw@mail.gmail.com>
 <FR0P281MB264974ECDCEB62EF7985DD9E92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnjdXYqh-C11tifsT2RjXV2xUcofE5bpZ7qV17QzsBWaqw@mail.gmail.com>
Message-ID: <FR0P281MB26496404F24209040F961F3F92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>

Ah, cool.

So that means:

Usually, at the end of the IFDs is the offset to the next IFD and Hamamatsu places all 8 instead of 4 bytes there and *thereafter* the high bits of all value/offset start, which would mean:

You have this IFD structure then:
-------------------------------------------

2 bytes:  for count of tags
12 bytes * (count of tags): The tags themselves
8 byte:  offset of next IFD
4 bytes * (count of tags): High bits for all tags

?

Thx

Martin

-----Original Message-----
From: Benjamin Gilbert <bgilbert+openslide at cs.cmu.edu> 
Sent: Freitag, 17. Mai 2024 19:30
To: openslide-users at lists.andrew.cmu.edu
Cc: Martin Weihrauch <m.weihrauch at smartinmedia.com>
Subject: Re: Question about NDPI

On Fri, May 17, 2024 at 10:01?AM Martin Weihrauch <m.weihrauch at smartinmedia.com> wrote:
> As the value/offset tags within the IFDs are only 4 bytes in 
> Hamamatsu, I saw from OpenSlide that
> - If it is not the 1st IFD, then it compares the offset of any tag with the offset to the tag from the 1st IFD. If the offset is the same --> take this 32bit offset, else combine the high bits of the 64bit offset of the IFD in the file with the tag 32 bit offset to 64bits.

It turns out that this heuristic is incorrect.  Nicholas Trahearn found that the high bits are actually present elsewhere in the file.
I haven't dug into this yet, but you can see his work at <https://github.com/openslide/openslide/pull/602>.  I'm hoping to switch OpenSlide to the new approach soon.

Best,
--Benjamin Gilbert

From m.weihrauch at smartinmedia.com  Fri May 24 15:43:31 2024
From: m.weihrauch at smartinmedia.com (Martin Weihrauch)
Date: Fri, 24 May 2024 19:43:31 +0000
Subject: Question about NDPI
In-Reply-To: <FR0P281MB26496404F24209040F961F3F92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
References: <CANioFng8nfWUxQXDfCGe8J89zAJCmC-eaDHMrcWF5LZxhSLqeQ@mail.gmail.com>
 <FR0P281MB2649DAEB3BA43CE232864E9892E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnj1tGMZn0XySCyinVoO9CjOu9GyaBnCceZ-YME+91sv0g@mail.gmail.com>
 <FR0P281MB2649050C26EA90894147192B92E32@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFniVz_dN5x-K=Mq=DUp80VR8LGa9c3x5cT_6Mf=CzchFhw@mail.gmail.com>
 <FR0P281MB264974ECDCEB62EF7985DD9E92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnjdXYqh-C11tifsT2RjXV2xUcofE5bpZ7qV17QzsBWaqw@mail.gmail.com>
 <FR0P281MB26496404F24209040F961F3F92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
Message-ID: <FR0P281MB2649F1EEA929256A6E26DCE492F52@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>

One question about the macro image in Hamamatsu's NDPI:

There is only one "associated image", the macro image, which has a source-lens of -1.
With the Hamamatsu SDK, you usually also get a label. I suppose, this is done within the SDK by cropping a part of the macro image?
Is there a certain percentage, etc of how much of the macro image is label? And is the label always on the left side?

Thank you 

Martin



From bgilbert+openslide at cs.cmu.edu  Fri May 24 15:59:34 2024
From: bgilbert+openslide at cs.cmu.edu (Benjamin Gilbert)
Date: Fri, 24 May 2024 14:59:34 -0500
Subject: Question about NDPI
In-Reply-To: <FR0P281MB2649F1EEA929256A6E26DCE492F52@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
References: <CANioFng8nfWUxQXDfCGe8J89zAJCmC-eaDHMrcWF5LZxhSLqeQ@mail.gmail.com>
 <FR0P281MB2649DAEB3BA43CE232864E9892E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnj1tGMZn0XySCyinVoO9CjOu9GyaBnCceZ-YME+91sv0g@mail.gmail.com>
 <FR0P281MB2649050C26EA90894147192B92E32@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFniVz_dN5x-K=Mq=DUp80VR8LGa9c3x5cT_6Mf=CzchFhw@mail.gmail.com>
 <FR0P281MB264974ECDCEB62EF7985DD9E92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnjdXYqh-C11tifsT2RjXV2xUcofE5bpZ7qV17QzsBWaqw@mail.gmail.com>
 <FR0P281MB26496404F24209040F961F3F92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <FR0P281MB2649F1EEA929256A6E26DCE492F52@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
Message-ID: <CANioFnhSTOQMcYHyv2zPDi+fLi0AnG0AnyKSv2ey+0iADdoZLQ@mail.gmail.com>

On Fri, May 24, 2024 at 2:43?PM Martin Weihrauch
<m.weihrauch at smartinmedia.com> wrote:
> There is only one "associated image", the macro image, which has a source-lens of -1.
> With the Hamamatsu SDK, you usually also get a label. I suppose, this is done within the SDK by cropping a part of the macro image?
> Is there a certain percentage, etc of how much of the macro image is label? And is the label always on the left side?

OpenSlide provides the data known to be present in the slide file, so
either the SDK is doing some postprocessing or we've missed something.
Does the SDK provide a label image for any of the NDPI slides in
openslide-testdata?

https://openslide.cs.cmu.edu/download/openslide-testdata/Hamamatsu/

Best,
--Benjamin Gilbert

From bgilbert+openslide at cs.cmu.edu  Tue May 28 05:46:35 2024
From: bgilbert+openslide at cs.cmu.edu (Benjamin Gilbert)
Date: Tue, 28 May 2024 04:46:35 -0500
Subject: Question about NDPI
In-Reply-To: <FR0P281MB26496404F24209040F961F3F92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
References: <CANioFng8nfWUxQXDfCGe8J89zAJCmC-eaDHMrcWF5LZxhSLqeQ@mail.gmail.com>
 <FR0P281MB2649DAEB3BA43CE232864E9892E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnj1tGMZn0XySCyinVoO9CjOu9GyaBnCceZ-YME+91sv0g@mail.gmail.com>
 <FR0P281MB2649050C26EA90894147192B92E32@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFniVz_dN5x-K=Mq=DUp80VR8LGa9c3x5cT_6Mf=CzchFhw@mail.gmail.com>
 <FR0P281MB264974ECDCEB62EF7985DD9E92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnjdXYqh-C11tifsT2RjXV2xUcofE5bpZ7qV17QzsBWaqw@mail.gmail.com>
 <FR0P281MB26496404F24209040F961F3F92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
Message-ID: <CANioFnhf2mDV7ODxZNhMm-BKAiZnM=A6KjMNwtuhpbwb-X5eOQ@mail.gmail.com>

On Fri, May 17, 2024 at 1:26?PM Martin Weihrauch
<m.weihrauch at smartinmedia.com> wrote:
> You have this IFD structure then:
> -------------------------------------------
>
> 2 bytes:  for count of tags
> 12 bytes * (count of tags): The tags themselves
> 8 byte:  offset of next IFD
> 4 bytes * (count of tags): High bits for all tags
>
> ?

Yes, that's correct.

Best,
--Benjamin Gilbert

From j.hudecek at nki.nl  Tue May 28 08:12:49 2024
From: j.hudecek at nki.nl (j.hudecek at nki.nl)
Date: Tue, 28 May 2024 12:12:49 +0000
Subject: Question about NDPI
In-Reply-To: <CANioFnhf2mDV7ODxZNhMm-BKAiZnM=A6KjMNwtuhpbwb-X5eOQ@mail.gmail.com>
References: <CANioFng8nfWUxQXDfCGe8J89zAJCmC-eaDHMrcWF5LZxhSLqeQ@mail.gmail.com>
 <FR0P281MB2649DAEB3BA43CE232864E9892E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnj1tGMZn0XySCyinVoO9CjOu9GyaBnCceZ-YME+91sv0g@mail.gmail.com>
 <FR0P281MB2649050C26EA90894147192B92E32@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFniVz_dN5x-K=Mq=DUp80VR8LGa9c3x5cT_6Mf=CzchFhw@mail.gmail.com>
 <FR0P281MB264974ECDCEB62EF7985DD9E92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnjdXYqh-C11tifsT2RjXV2xUcofE5bpZ7qV17QzsBWaqw@mail.gmail.com>
 <FR0P281MB26496404F24209040F961F3F92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>,
 <CANioFnhf2mDV7ODxZNhMm-BKAiZnM=A6KjMNwtuhpbwb-X5eOQ@mail.gmail.com>
Message-ID: <84bfaffbc1c14e6784247ffde1621cb6@nki.nl>

Hi Martin,

Are you making a new slide-reading library that will also support NDPI? Very cool!
Will you also open-source it? Sounds like it could be considered derived work.

Regards,
Jan Hudecek
________________________________________
From: openslide-users <openslide-users-bounces+j.hudecek=nki.nl at lists.andrew.cmu.edu> on behalf of Benjamin Gilbert <bgilbert+openslide at cs.cmu.edu>
Sent: Tuesday, May 28, 2024 11:46:35 AM
To: openslide-users at lists.andrew.cmu.edu
Subject: Re: Question about NDPI

LET OP: Deze e-mail is afkomstig van buiten de organisatie. Open alleen links of bijlagen als je de afzender kent en weet dat de inhoud veilig is.
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

On Fri, May 17, 2024 at 1:26?PM Martin Weihrauch
<m.weihrauch at smartinmedia.com> wrote:
> You have this IFD structure then:
> -------------------------------------------
>
> 2 bytes:  for count of tags
> 12 bytes * (count of tags): The tags themselves
> 8 byte:  offset of next IFD
> 4 bytes * (count of tags): High bits for all tags
>
> ?

Yes, that's correct.

Best,
--Benjamin Gilbert
_______________________________________________
openslide-users mailing list
openslide-users at lists.andrew.cmu.edu
https://lists.andrew.cmu.edu/mailman/listinfo/openslide-users

From m.weihrauch at smartinmedia.com  Tue May 28 11:25:04 2024
From: m.weihrauch at smartinmedia.com (Martin Weihrauch)
Date: Tue, 28 May 2024 15:25:04 +0000
Subject: Question about NDPI
In-Reply-To: <84bfaffbc1c14e6784247ffde1621cb6@nki.nl>
References: <CANioFng8nfWUxQXDfCGe8J89zAJCmC-eaDHMrcWF5LZxhSLqeQ@mail.gmail.com>
 <FR0P281MB2649DAEB3BA43CE232864E9892E02@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnj1tGMZn0XySCyinVoO9CjOu9GyaBnCceZ-YME+91sv0g@mail.gmail.com>
 <FR0P281MB2649050C26EA90894147192B92E32@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFniVz_dN5x-K=Mq=DUp80VR8LGa9c3x5cT_6Mf=CzchFhw@mail.gmail.com>
 <FR0P281MB264974ECDCEB62EF7985DD9E92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>
 <CANioFnjdXYqh-C11tifsT2RjXV2xUcofE5bpZ7qV17QzsBWaqw@mail.gmail.com>
 <FR0P281MB26496404F24209040F961F3F92EE2@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>,
 <CANioFnhf2mDV7ODxZNhMm-BKAiZnM=A6KjMNwtuhpbwb-X5eOQ@mail.gmail.com>
 <84bfaffbc1c14e6784247ffde1621cb6@nki.nl>
Message-ID: <FR0P281MB2649C81C379D46D94BAB538192F12@FR0P281MB2649.DEUP281.PROD.OUTLOOK.COM>

Thank you for your interest!

Actually, I think there is no need to develop an alternative for OpenSlide. However, I am writing an anonymizer for WSI, which will be completely in .NET / C#. Basically, OpenSlide without the tile reading part.
Hamamatsu is now fully implemented for anonymization of label, metadata, etc. Working on the next ones.

Unfortunately, I have no plans yet to open source it / give it away for free, because our company also has to pay employees and bills, but we'll see, maybe I'll find a fair model.

Best

Martin




-----Original Message-----
From: openslide-users <openslide-users-bounces+m.weihrauch=smartinmedia.com at lists.andrew.cmu.edu> On Behalf Of j.hudecek at nki.nl
Sent: Dienstag, 28. Mai 2024 14:13
To: openslide-users at lists.andrew.cmu.edu
Subject: Re: Question about NDPI

[Sie erhalten nicht h?ufig E-Mails von j.hudecek at nki.nl. Weitere Informationen, warum dies wichtig ist, finden Sie unter https://aka.ms/LearnAboutSenderIdentification ]

Hi Martin,

Are you making a new slide-reading library that will also support NDPI? Very cool!
Will you also open-source it? Sounds like it could be considered derived work.

Regards,
Jan Hudecek
________________________________________
From: openslide-users <openslide-users-bounces+j.hudecek=nki.nl at lists.andrew.cmu.edu> on behalf of Benjamin Gilbert <bgilbert+openslide at cs.cmu.edu>
Sent: Tuesday, May 28, 2024 11:46:35 AM
To: openslide-users at lists.andrew.cmu.edu
Subject: Re: Question about NDPI

LET OP: Deze e-mail is afkomstig van buiten de organisatie. Open alleen links of bijlagen als je de afzender kent en weet dat de inhoud veilig is.
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

On Fri, May 17, 2024 at 1:26?PM Martin Weihrauch <m.weihrauch at smartinmedia.com> wrote:
> You have this IFD structure then:
> -------------------------------------------
>
> 2 bytes:  for count of tags
> 12 bytes * (count of tags): The tags themselves
> 8 byte:  offset of next IFD
> 4 bytes * (count of tags): High bits for all tags
>
> ?

Yes, that's correct.

Best,
--Benjamin Gilbert
_______________________________________________
openslide-users mailing list
openslide-users at lists.andrew.cmu.edu
https://lists.andrew.cmu.edu/mailman/listinfo/openslide-users
_______________________________________________
openslide-users mailing list
openslide-users at lists.andrew.cmu.edu
https://lists.andrew.cmu.edu/mailman/listinfo/openslide-users