Create a 64bit build environment for the MinGW.org project.
Changes to WSL for this milestone will occur in branch 6.0-exp. All changes are to be discussed via a new ticket. We are moving w32api directory to wslapi directory and using a template approach for the library definition files.
The following conversation was transcribed from the developer's Slack channel, with Keith's permission.
It is transcribed inline, for better reference, while obeying the 2000 character per comment limit on this tracker.
Cesar Strauss Mar 23rd, 2019 at 9:48 AM
I restarted the work on porting the MinGW Runtime to 64 bits.
Cesar Strauss Mar 23rd, 2019 at 10:25:23 AM
Currently I'm investigating a failure in pseudo-reloc.c.
Cesar Strauss Mar 24th, 2019 at 5:53:58 AM
Actually, pseudo-reloc.c seems to work fine.
Cesar Strauss Mar 24th, 2019 at 9:11:36 PM
The issue was just an API call which was declared with an argument of the wrong size (should be SIZE_T instead of DWORD).
Cesar Strauss Mar 24th, 2019 at 9:18:05 PM
There is still work to do, but I can already create working executables that don't simply crash at startup.
Cesar Strauss Mar 24th, 2019 at 9:36:21 AM
I think I can proceed to write tickets on OSDN, with preliminary patches, for discussion.
Keith Mar 28th, 2019 at 11:16:13 AM
@Cesar Strauss
I think I can proceed to write tickets on OSDN, with preliminary patches, for discussion.
That does seem like the way to go. Do, please, be aware that I still have a patch queue, of around 130 outstanding patches, to address (mostly self-containment) issues in WINAPI; it would be good if we can minimize potential conflicts.
Keith Mar 28th, 2019 at 3:03:58 PM
The issue was just an API call which was declared with an argument of the wrong size (should be SIZE_T instead of DWORD).
Yuck! I really don't like these ghastly Microsoft "invented" data type names, particularly where they are just ugly aliases for a perfectly good standard data type; what's wrong with size_t (defined in <stddef.h>, which is already included anyway)?
That said, I guess that, for code which is dependent on Windows APIs, there may be some justification for using them, in this context. I do understand that Microsoft's motivation for creating them was as a portability feature, to facilitate porting (originally) from 16-bit to 32-bit Windows. However, gratuitously over-using them may actually harm portability, and DWORD may be particularly harmful! For example, when I ported pexports (which doesn't actually depend on any WinAPI functions) to run on 64-bit Linux, I found it to be littered with Microsoft "invented" data types, and DWORD, which should be equivalent to the standard uint32_t, but due to its definition in terms of a non-fixed-width integer type, although still a 32-bit type on Win-64, is actually interpreted as 64-bit on 64-bit Linux. This utterly destroyed the layout of structures used to map DLL internals, in pexports, thus leading to runtime crashes; the solution was to replace all references to type DWORD with the correctly sized fixed-width uint32_t standard (and infinitely more portable) data type.
Earnie Boyd Mar 30th, 2019 at 3:03:58 PM
That does seem like the way to go. Do, please, be aware that I still have a patch queue, of around 130 outstanding patches, to address (mostly self-containment) issues in WINAPI; it would be good if we can minimize potential conflicts.
Perhaps do the github method of forking the source to a local "Chamber" and submit merge requests?
Keith Mar 30th, 2019 at 6:16:03 PM
Perhaps do the github method of forking the source to a local "Chamber" and submit merge requests?
I could do that, but it's a Mercurial patch queue, so it would need to be a Mercurial chamber; are you up for working with that?
Cesar Strauss Mar 30th, 2019 at 6:21:29 PM
Sure.
Cesar Strauss Mar 30th, 2019 at 6:26:03 PM
I noticed there is a 6.0-exp branch for Win64 support, do you want maybe to build on this?
Cesar Strauss Mar 31st, 2019 at 9:51:45 PM
@Keith, as I understand it, you would share your Mercurial repository, so I could clone it, right?
Keith Mar 31st, 2019 at 10:14:18 AM
@Cesar Strauss
as I understand it, you would share your Mercurial repository, so I could clone it, right?
No, you can clone the existing git repository, using Mercurial's hg-git extension, to create a Mercurial clone, (as I have it here). I would publish a Mercurial repository, containing just my patch queue, which you would then clone into the .hg/patches directory of your Mercurial clone of the git repository. You may then use the Mercurial queues feature, to qpush, qrefresh, and/or qpop patches within your Mercurial clone of the git repository.
If you modify any of the patches, you would send me a diff of the queue, generated by hg diff --mq, so I can keep our patches in sync.
Keith Mar 31st, 2019 at 11:49:32 AM
@Cesar Strauss
I would publish a Mercurial repository, containing just my patch queue
That's now done; you will find it at https://osdn.net/users/keith/pf/mingw-wsl-patches/
Keith Mar 31st, 2019 at 2:50:32 PM
@Cesar Strauss
I noticed there is a 6.0-exp branch for Win64 support, do you want maybe to build on this?
That would seem sensible; after all, that branch was created expressly for the purpose of working towards Win64 support. Do note, however, that there have been significant advances on the 5.x-trunk branches, (culminating in 5.2-trunk), since the most recent activity on 6.0-exp; the tip state of 5.2-trunk should be merged into 6.0-exp, before it progresses further.
Keith Apr 1st, 2019 at 7:23:33 AM
@Cesar Strauss, in addition to the patch queue, which I've already published, I have a couple of Win64-specific patches (in an independent queue). I've never been able to test these, since I don't have access to a Win64 host, (and VirtualBox will not let me virtualize one, on my P6200 64-bit Linux host).
How would you prefer me to make these available? Another OSDN.net chamber? Or attach them to @Earnie Boyd's existing OSDN.net Win64 ticket? Or even, just apply and push, on the 6.0-exp branch?
Keith Apr 1st, 2019 at 10:35:56 AM
@Cesar Strauss, I forgot to mention that, in my working copy, I currently have the self-contained patch queue guard enabled: hg qselect self-contained.
Keith Apr 1st, 2019 at 12:31:02 PM
@Cesar Strauss
the tip state of 5.2-trunk should be merged into 6.0-exp, before it progresses further.
I've done that for you, now.
Keith Apr 1st, 2019 at 6:28:18 PM
@Cesar Strauss, as I originally published it, mingw-wsl-patches will not apply on the 6.0-exp branch, (due to the rename of the w32api directory). To address this, I've modified all the affected path name references, within all of the patch files, and posted an update to a new 6.0-exp bookmark, (bookmarks are Mercurial's equivalent of git's branches; git has no analogue for Mercurial's branches), within mingw-wsl-patches. I've also set a 5.2-trunk bookmark, representing the originally published state of the patch queue.
After cloning mingw-wsl-patches into the .hg/patches directory of your Mercurial clone of the git repository, you will need to update it to the appropriate bookmark, to match the git branch (which will also appear as a bookmark in your Mercurial clone), on which you are working.
Cesar Strauss Apr 1st, 2019 at 10:24:17 PM
@Cesar Strauss, in addition to the patch queue, which I've already published, I have a couple of Win64-specific patches (in an independent queue). I've never been able to test these, since I don't have access to a Win64 host, (and VirtualBox will not let me virtualize one, on my P6200 64-bit Linux host).
I will be glad to test them.
How would you prefer me to make these available? Another OSDN.net chamber?
Sure.
Or attach them to @Earnie Boyd's existing OSDN.net Win64 ticket?
Not necessarily. We could create new tickets.
Maybe, instead of attaching patches, add a web link to the changeset, provided by the OSDN webview.
Or even, just apply and push, on the 6.0-exp branch?
Let me test them first, thanks.
Keith Apr 2nd, 2019 at 7:17:37 PM
@Cesar Strauss
How would you prefer me to make these available? Another OSDN.net chamber?
When I said "a couple of patches", I meant that literally ... there are just two of them! Rather than create a new chamber, for only two patches, I've simply added them to my existing mingw-wsl-patches chamber, on the Win64-specific 6.0-exp bookmark. They appear as the first two in the series, when both the working directory and the queue are on the 6.0-exp bookmark, and both bear +win64 guards.
Cesar Strauss Apr 20th, 2019 at 7:37:25 PM
Here is my first patch: https://osdn.net/users/cstrauss/pf/mingw-wsl-patches-win64/scm/blobs/tip/win64-context.patch
Cesar Strauss Apr 20th, 2019 at 7:44:41 PM
It is a workaround to the first failure that occurs as I build for 64 bits.
Cesar Strauss Apr 20th, 2019 at 7:45:57 PM
The next patches will continue to be in the order in which they fix build problems.
Cesar Strauss Apr 20th, 2019 at 8:12:04 PM
Next failure is in cpu_features.sx. In this case, one of @Keith's patches helps.
Keith Apr 21st, 2019 at 7:19:35 AM
Next failure is in cpu_features.sx. In this case, one of @Keith's patches helps.
Yeah, assembly code is necessarily architecture specific, and I wrote the original for ix86; I proposed that patch, in recognition that x86-64 would need different coding. Was my patch, alone, sufficient? Or did you need to rework it?
Keith Apr 21st, 2019 at 7:41:50 AM
Here is my first patch: https://osdn.net/users/cstrauss/pf/mingw-wsl-patches-win64/scm/blobs/tip/win64-context.patch
Should you not, rather, forward declare struct _CONTEXT, with CONTEXT being typedef struct _CONTEXT CONTEXT? Doesn't https://docs.microsoft.com/en-gb/windows/desktop/api/winnt/ns-winnt-context then provide the details of the structure declaration you need?
Keith Apr 21st, 2019 at 8:00:24 AM
While we're discussing architecture specific CONTEXT declarations, some time ago @Earnie Boyd suggested factoring the individual declarations out of <winnt.h>, into architecture specific individual files. That seems to me to be a very reasonable suggestion; maybe we should start by placing the _WIN64 specific declaration in its own stand-alone header file?
Cesar Strauss Apr 21st, 2019 at 8:17:45 AM
Was my patch, alone, sufficient? Or did you need to rework it?
I just had to correct one missing instance of ebx->rbx.
Cesar Strauss Apr 21st, 2019 at 8:27:45 AM
I guess I overlooked this. I will now update the patch to use the reference CONTEXT declaration from MSDN.
Cesar Strauss Apr 21st, 2019 at 8:30:04 AM
Next failure is in cpu_features.sx. In this case, one of @Keith's patches helps.
This file now builds, but I cannot properly test it until the compiler is able to generate executables, which is later in the patch series.
Cesar Strauss Apr 21st, 2019 at 9:52:28 AM
Updated "cpu-features" patch at https://osdn.net/users/cstrauss/pf/mingw-wsl-patches-win64/scm/blobs/tip/cpu-features-x86-64.patch
Cesar Strauss Apr 21st, 2019 at 1:00:56 PM
Doesn't https://docs.microsoft.com/en-gb/windows/desktop/api/winnt/ns-winnt-context then provide the details of the structure declaration you need?
It depends on missing declarations: XMM_SAVE_AREA32, NEON128 and M128A. I can't find them in MSDN.
Cesar Strauss Apr 21st, 2019 at 2:24:42 PM
Should you not, rather, forward declare struct _CONTEXT, with CONTEXT being typedef struct _CONTEXT CONTEXT?
New version of the patch at https://osdn.net/users/cstrauss/pf/mingw-wsl-patches-win64/scm/blobs/3659981a237068fbb53a235468d380237bfa40eb/win64-context.patch
Cesar Strauss Apr 21st, 2019 at 3:12:32 PM
maybe we should start by placing the _WIN64 specific declaration in its own stand-alone header file?
Proposed patch at https://osdn.net/users/cstrauss/pf/mingw-wsl-patches-win64/scm/blobs/f9c7f78569e48728c628ab0157ddbcc3821089d7/winnt-decl-64.patch
Keith Apr 22nd, 2019 at 7:35:04 AM
@Cesar Strauss
I just had to correct one missing instance of ebx->rbx.
Clearly, an oversight on my part; thanks for catching it!
Keith Apr 23rd, 2019 at 5:21:21 AM
@Cesar Strauss, I've pulled current state from your mingw-wsl-patches-win64 fork, into my own mingw-wsl-patches. I've also applied your modified cpu-features-x86-64.patch on to the tip of my 5.2-trunk development branch, and reassembled for _WIN32; it generates identical object code to that from the original _WIN32 only version, so that's looking good for _WIN32 releases. Do you still want some extra time to verify it for _WIN64, before I hg qfinish it?
Cesar Strauss Apr 23rd, 2019 at 7:39:59 PM
Yes, I need a little more time to verify its behaviour at runtime. I'll quickly forward port the remaining patches, create a compiler that can generate executables, and run test programs.
Cesar Strauss Apr 23rd, 2019 at 10:48:08 AM
There's another issue: on Win64, C symbols don't get a leading underscore in Assembly.
Keith Apr 24th, 2019 at 1:10:40 PM
on Win64, C symbols don't get a leading underscore in Assembly.
I think I knew this, but failed to fully appreciate its significance :white_frowning_face: This is actually a rather annoying inconsistency; it means that every assembly language module will need to be reviewed, and those which are currently bare .s may need to become .sx, so that the C preprocessor can handle the symbolic discrepancy.
Before we embark on this, in earnest, I've pushed a refresh of cpu-features-win64.patch; does it address the issue adequately, for this particular case?
Keith Apr 25th, 2019 at 7:08:20 AM
@Cesar Strauss
depends on missing declarations: XMM_SAVE_AREA32, NEON128 and M128A. I can't find them in MSDN.
Neither can I; typical of Microsoft's woeful failure to provide adequate documentation:white_frowning_face: However, I did find these (under BSD, or X-Open/MIT licences):
http://winappdbg.sourceforge.net/doc/v1.3/winappdbg.win32.context_amd64-pysrc.html#XMM_SAVE_AREA32
http://winappdbg.sourceforge.net/doc/v1.3/winappdbg.win32.defines-pysrc.html#M128A
https://github.com/dotnet/coreclr/blob/master/src/inc/crosscomp.h
Keith Apr 25th, 2019 at 5:57:45 PM
on Win64, C symbols don't get a leading underscore in Assembly.
That's actually the least of the potential problems with assembly language implementations; more significant, is the change in function calling conventions, and in particular, the argument passing protocol: https://docs.microsoft.com/en-gb/cpp/build/x64-calling-convention?view=vs-2017
Cesar Strauss Apr 25th, 2019 at 9:07:36 PM
Before we embark on this, in earnest, I've pushed a refresh of cpu-features-win64.patch; does it address the issue adequately, for this particular case?
Yes, it does.
Cesar Strauss Apr 26th, 2019 at 6:03:27 AM
I'll also look at the assembly output of the compiler to better understand the calling conventions.
Keith Apr 26th, 2019 at 3:50:53 PM
I'll also look at the assembly output of the compiler to better understand the calling conventions.
That's how I initially became aware of the problem — looking at the -S output from a stage one build of a Linux hosted mingw64 cross-compiler. It's instructive, but it needs some further reading for adequate understanding. So far, I've deduced:
keith)
Fork
This is the primary ticket for controlling this milestone. Status of the milestone will be added as comments to this ticket.