That library was reworked by me in 2008, before LabVIEW had any 64-bit version. Unfortunately, SOCKET in Windows is an UINT_PTR which means that it is a 32-bit integer on 32-bit LabVIEW and 64-bit integer on 64-bit LabVIEW. But you can NOT port this over to Linux as is. There the file-descriptor used for all socket functions is an explicit int, so always a 32-bit value! This is because the Berkeley socket library is originally developed for Unix and build around the generic socket concept in Unix which traditionally uses int file descriptors. When Microsoft took the Berkeley socket library and adapted it for Windows as WinSock library, it was already very revolutionary to have this as a fully 32-bit code library, Windows itself was still mainly a selector based 16-bit environment. And Microsoft likes to use handles, which are opaque pointers and happened to be 32-bit integers too back then.
The library as posted on the NI forum does NOT work in 64-bit Windows, and that is not just because of the 64-bit SOCKET handle itself but also because of the fd_set data structure which is defined as follows:
typedef struct fd_set {
u_int fd_count; /* how many are SET? */
SOCKET fd_array[FD_SETSIZE]; /* an array of SOCKETs */
} fd_set;
Now on 32-bit LabVIEW there is nothing strange here. The fd_array with SOCKET handles begins at offset 4, so the LabVIEW array of 2 * 32-bit integers works perfectly fine. The first array element corresponds to fd_count and the second element to the first element in the fd_array and if fd_count doesn't contain more than 1 as value it does not matter that the fd_array doesn't contain all 64 elements that this structure is declared for.
On 64-bit the SOCKET is a 64-bit integer and is naturally aligned on an 8 byte boundary in that structure. So there is a dummy 32-bit filler element between fd_count and fd_array. And the first SOCKET should be a 64-bit integer.
Solution to make this work on 64-bit LabVIEW, besides changing the SOCKET handle parameter for all functions to be a pointer sized integer, is to use a conditional compile structure. For the 32-bit case use an array of 32-bit integers as fd_set just as is used now, for the 64-bit case you need to use an array of 64-bit integers. The rest remains the same. You even can use an array of 2 * 64-bit integers, just the same as for the 32-bit case with an array of 32-bit integers. Since we run on a Little Endian machine under Windows, the lower significant 32-bits of the first 64-bit element happen to match the location in memory where the fd_count is expected, the extra 4 filler bytes are overwritten by the higher significant 32-bits of the first 64-bit array element which is not a problem. They are really DON'T CARE.
Your observation that this weird error can happen if the fd_set structure is not correctly initialized was spot on. But not because it is not really correctly initialized in the VI (it is) but because the array of 32-bit values has a different layout than what the 64-bit Winsock library expects.