Building Prime95 ASM on Linux WIP

[M344587487]

I'm playing with UASM and the makefiles in Prime95 with the goal of being able to build factor32/factor64 and eventually gwnum from scratch using Linux. Hopefully with a bit of sed magic the existing makefiles can be automatically converted to makefiles that work under Linux, or at least get us 95% of the way there. This thread is to help debug issues encountered along the way.

To follow along, objconv and uasm need to be in the PATH. objconv compiles for Linux easily (http://www.agner.org/optimize/#objconv ), UASM is more of a pain (https://github.com/Terraspace/UASM ). UASM v2.50 compiled with gccLinux64.mak after adding the -fPIE option to the build process. To compile newer versions the fix for direct.h would be to port it using unistd or a wrapper, but there are other errors stopping the compile process with non-obvious fixes so v2.50 will do for now. This script executed from the UASM repo should build UASM:

#!/bin/bash
git switch v2.50
sed -i 's/extra_c_flags =/extra_c_flags = -fPIE/g' gccLinux64.mak
make -f gccLinux64.mak
cp GccUnixR/uasm uasm

So far the C compiler and linker for gwnum haven't been touched but progress has been made with UASM. Object files are generated for all ASM in the four makefiles (in gwnum by deleting the currently broken C and library targets) except for these six targets in gwnum/compil64: amd64/zr4dwpn_skx.obj amd64/zr4dwpn2_skx.obj amd64/timeit.obj linux64/zr4dwpn_skx.obj linux64/zr4dwpn2_skx.obj linux64/timeit.obj

The error messages for the failing *skx.obj's are similar to this:

uasm -c -DX86_64 -DWINDOWS64 -win64 -DARCH=SKX -Foamd64/zr4dwpn_skx.obj zr4dwpn.asm
UASM v2.50, Aug 19 2021, Masm-compatible assembler.
Portions Copyright (c) 1992-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.

zr4dwpn.asm(26) : Error A2169: General Failure
 push_reg(2)[macamd64.inc]: Macro called from
  ah_prolog(64)[unravel.mac]: Macro called from
   ad_prolog(58)[unravel.mac]: Macro called from
    zfft_header(2)[zmult.mac]: Macro called from
     zr4dwpn_pass1sc2(2)[zr4dwpnpass1sc.mac]: Macro called from
      zpass1gen(67)[zmult.mac]: Macro called from
       buildfor(8)[zarch.mac]: Macro called from
        zr4dwpn.asm(26): Main line code
make: *** [compil64_linux:282: amd64/zr4dwpn_skx.obj] Error 1

And the failing timeit.obj's are similar to this:

uasm -c -DX86_64 -DWINDOWS64 -win64 -DARCH=FMA3 -Foamd64/timeit.obj timeit.asm
UASM v2.50, Aug 19 2021, Masm-compatible assembler.
Portions Copyright (c) 1992-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.

timeit.asm(2109) : Error A2302: Not possible to EVEX encode requested instruction
 read8(2)[timeit.asm]: Macro called from
  timeit.asm(2109): Main line code
timeit.asm: 2343 lines, 1 passes, 528673 ms, 0 warnings, 1 errors
 make: *** [compil64_linux:306: amd64/timeit.obj] Error 1

uasm64.exe through wine also fails to assemble these objects, although interestingly (maybe because it's v2.52) comes up with different error messages:

UASM v2.52, Apr  2 2021, Masm-compatible assembler.
Portions Copyright (c) 1992-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.

zr4dwpn.asm(40) : Error A2014: invalid combination of opcode and oper
ands
 L1prefetchw(3)[zbasics.mac]: Macro called from
  zr16_16c_djbunfft_cmn(216)[zr16.mac]: Macro called from
   zr16_sixteen_complex_djbunfft(1)[zr16.mac]: Macro called from
    zr4dwpn_pass1sc2048(108)[zr4dwpnpass1sc.mac]: Macro called from
     zpass1gen(67)[zmult.mac]: Macro called from
      buildfor(8)[zarch.mac]: Macro called from
       build421(3)[zarch.mac]: Macro called from
        zr4dwpn.asm(40): Main line code
zr4dwpn.asm(42) : Error A2014: invalid combination of opcode and oper
ands
 L1prefetchw(3)[zbasics.mac]: Macro called from
  zr16_16c_djbunfft_cmn(216)[zr16.mac]: Macro called from
   zr16_sixteen_complex_djbunfft(1)[zr16.mac]: Macro called from
    zr4dwpn_pass1sc3072(108)[zr4dwpnpass1sc.mac]: Macro called from
     zpass1gen(67)[zmult.mac]: Macro called from
      buildfor(8)[zarch.mac]: Macro called from
       build421(3)[zarch.mac]: Macro called from
        zr4dwpn.asm(42): Main line code
zr4dwpn.asm(60) : Error A2014: invalid combination of opcode and oper
ands
 L1prefetchw(3)[zbasics.mac]: Macro called from
  zr16_16c_djbunfft_cmn(216)[zr16.mac]: Macro called from
   zr16_sixteen_complex_djbunfft(1)[zr16.mac]: Macro called from
    zr4dwpn_pass1sc2048ac(89)[zr4dwpnpass1sc.mac]: Macro called from
     zpass1gen(67)[zmult.mac]: Macro called from
      buildfor(8)[zarch.mac]: Macro called from
       build421(3)[zarch.mac]: Macro called from
        zr4dwpn.asm(60): Main line code
zr4dwpn.asm(62) : Error A2014: invalid combination of opcode and oper
ands
 L1prefetchw(3)[zbasics.mac]: Macro called from
  zr16_16c_djbunfft_cmn(216)[zr16.mac]: Macro called from
   zr16_sixteen_complex_djbunfft(1)[zr16.mac]: Macro called from
    zr4dwpn_pass1sc3072ac(89)[zr4dwpnpass1sc.mac]: Macro called from
     zpass1gen(67)[zmult.mac]: Macro called from
      buildfor(8)[zarch.mac]: Macro called from
       build421(3)[zarch.mac]: Macro called from
        zr4dwpn.asm(62): Main line code
zr4dwpn.asm: 65 lines, 1 passes, 1043 ms, 0 warnings, 4 errors
00f4:fixme:ver:GetCurrentPackageId (000000000021FDA0 0000000000000000): stub
make: *** [compil64_linux:282: amd64/zr4dwpn_skx.obj] Error 1

I can't read ASM however the offending lines in timeit.asm appear to be AVX512-related which may be relevant (also my machine does not have AVX512 but surely that's irrelevant?):

av512case0:    read8    4096, 16384    ; Read 4KB

All the generated object files are so far untested. As for the compiler and linker, I'm guessing there's enough differences between cl/gcc and lib/ld (lib is the equivalent of ld right?) that we'd be as well to manually create the build targets.

Here's the current (incomplete/buggy) state of the conversion script that at least works with the factor* makefiles and most of the ASM targets of the gwnum makefiles:

#!/bin/bash
cp $1 $1_linux
#Rename programs
sed -i 's/masm=uasm64/masm=uasm/g' $1_linux
sed -i 's/masm64=uasm64/masm64=uasm/g' $1_linux
sed -i 's/masm64lx=uasm64/masm64lx=uasm/g' $1_linux
sed -i 's/objconv=c:\\objconv\\objconv/objconv=objconv/g' $1_linux
sed -i 's/\\strip/\\stripper/g' $1_linux
sed -i 's/ strip/ .\\stripper/g' $1_linux
sed -i 's/ attrib/ chmod/g' $1_linux
sed -i 's/ del \/q/ rm -r/g' $1_linux
sed -i 's/ del/ rm -r/g' $1_linux
sed -i 's/ copy -q/ cp/g' $1_linux
sed -i 's/ copy/ cp/g' $1_linux
sed -i 's/ xcopy -y/ xcopy/g' $1_linux
sed -i 's/ xcopy -q/ xcopy/g' $1_linux
sed -i 's/ xcopy/ cp/g' $1_linux
sed -i 's/    rmdir macosx\\foo//g' $1_linux

#replace line breaks with placeholder
sed -i 's/\\$/LINECONTINUATION/g' $1_linux
#Change option delimiter from /to -
sed -i 'sq/q-qg' $1_linux
#Change path delimiter from \to /
sed -i 's/\\/\//g' $1_linux

#four spaces for tabs
sed -i 's/    /\t/g' $1_linux

#Restore line breaks
sed -i 's/LINECONTINUATION/\\/g' $1_linux
#sed -i 's///g' $1_linux

And this file "stripper" needs to be in the same directory as strip.bat:

#!/bin/bash
#bash equivalent of strip.bat, assumes objconv is in PATH
#Not named strip to avoid confusion with strip,
#and potential call-clashing on windows with strip.bat

#If no output file is supplied to uasm with -Fo, the default
#extension on windows is apparently .obj and .o on Linux
#The following if statement is a hack to rename
#non-existent .obj files passed to stripper from .o to .obj
if [ ! -f $1 ];then
    actualname=${1:0:${#1}-2}
    mv $actualname $1
fi
objconv -felf $1 dummy.o
objconv -fcoff dummy.o $1
rm dummy.o

All may become clear with enough sharp sticks to poke this problem with, might give it another crack at the weekend.

[Prime95]

I use Uasm 2.48. I reported a bug about 2.49, not sure if it got fixed. I don't even remember what the problem was. Consult the UASM bug database.

timeit.asm is not at all important. I use it to time snipets of assembly code. Commenting out offending lines of source code ought to get you past this blockage.

I objconv needed? Can UASM output ELF directly? Or perhaps it is safer to let UASM output COFF format and use objconv. I vaguely recall that objconv does some name mangling, so objconv may be required.

[M344587487]

Thanks for pointing to v2.48, I can't compile or find a Linux binary of v2.48 but am successfully using the windows version under wine to build gwnum. UASM definitely has cross-version and cross-OS issues which is annoying, for some reason v2.48 can compile gwnum but fails to build factor32 and factor64 and there's peculiar differences between Ubuntu20 and CentOS7 that need to be tested.

Initially tried a 1:1 conversion using objconv for ease and safety but it turns out it's much easier to scrap all the binary conversion targets completely. objconv in the standard build process converts to Linux/mac but is also used to remove debug symbols in windows .obj files (in strip.bat) to work around a MSVC issue. It doesn't appear to be necessary in the port, the port does have UASM output elf directly.

So far CentOS7 has managed to build gwnum64 using UASM v2.48 under wine and gcc. mprime was successfully built with it and appears to function so that's a good step, but we're a way off a fully native compile yet. The scripts are being cleaned and will be tested across many UASM versions to see what's what, if they appear to be in good shape they'll get posted later hopefully along with some UASM progress.

[M344587487]

Here's some build results with different uasm versions with 30.3b6, uasmw means the windows binary wrapped with wine and the number is the version. Earlier I thought factor64 was successfully assembled with something but I'd accidentally tested with an altered asm file.

1 for successfully built on CentOS 7, 0 for fail:

CentOS7 factor32 factor64 gwnum32 gwnum64
uasm245       1        0       0       0
uasm250       1        0       1       0
uasm254       0        0       0       0
uasmw248      0        0       1       1
uasmw252      1        0       1       0

Ubuntu 20.04 was the same except uasm245 also successfully built gwnum32. The uasm binaries used are the same on both OS's so the difference is weird, at least the failures/successes have so far been repeatable. uasm245 and uasm254 were found online (v2.54 is from an experimental branch and v2.45 is the newest version available from the main site). George can you definitely successfully assemble factor32 and factor64 with v2.48? UASM is a strange one.

I have no confidence that 32 bit anything is actually built properly (or 64 bit for that matter), all success means is that every target in the makefile completed so all expected object files exist.

Attached are the scripts in their current state, and an (entirely at own risk) mprime binary built on CentOS 7 using gwnum64 built from scratch and the factor64 object supplied with the source. It seems to work but hasn't been tested thoroughly. To use the scripts put them in the root directory, see readme. To build the 64 bit binary as attached you run the build_gwnum64 script (using appropriate uasm location) from the root directory then build gwnum and mprime as normal with gwnum/make64 and linux64/make.

[chalsall]

Originally Posted by M344587487

Attached are the scripts in their current state...

Thanks for taking on this non-trivial problem.

Many people don't appreciate that, like all engineering disciplines, there can be cases where the "tool-chain" is lost and/or only runs in antiquated environments.

I read a news article several months ago about the USAF having to hire a reverse engineering firm to be able to duplicate a critical part on some very expensive flight kit (an SR-71 maybe; I can't find the article at the moment). They had not only lost the tools need to build the component with (probably long scrapped), but even the blueprints.

[Prime95]

Factor32 assembles with 2.49
Factor64 does not assemble with either 2.48 or 2.49

[Prime95]

Factor64 "fix":

;;  UASM BUG	YMM_ONE			DD	1,0,1,0,1,0,1,0
YMM_ONE			DD	1
REST_OF_YMM_ONE			DD	0,1,0,1,0,1,0

I've submitted a bug report to UASM developers

[M344587487]

Thanks that's very instructive to a complete novice, suddenly half the code isn't black magic.

./build_factor64 "uasm245" build status: Failure
./build_factor64 "uasm250" build status: Success
./build_factor64 "uasm254" build status: Failure
./build_factor64 "uasmw248" build status: Success
 ./build_factor64 "uasmw252" build status: Success

It compiles on 3 of the 5 now, the important 3 so that's great (pretty sure uasm254 is a dud and uasm245 is old).