From diy-linux-dev@diy-linux.org Sat Oct 27 06:48:43 2007 From: diy-linux-dev@diy-linux.org (Greg Schafer) Date: Sat, 27 Oct 2007 16:48:43 +1000 Subject: Multilib build method Message-ID: <20071027064843.GA8402@eyo32.local> Hi Guys, Sorry for lack of activity lately. This is just a headsup about some stuff I've been working on. Alexander Patrakov prodded me into looking at the ideas presented here: http://www.mail-archive.com/lfs-dev@linuxfromscratch.org/msg10379.html The idea is not really that new, but now that I have it working I have it to say - it's bloody fantastic :-) The basic idea is to produce the current Binutils Pass 1 and GCC Pass 1 as cross tools. Then use the cross toolchain to cross compile Glibc. Once Glibc is in place, then build native tools against the Glibc and carry on as normal. The amazing thing is it actually *simplifies* a lot of stuff as compared to the current build! But the best part of all - it enables *sane* x86_64 multilib!! And when I say "sane" I mean understandable by mere mortals and with minimal hackery. And it doesn't really matter that much how sane the host is, or whether the host is 32 or 64 bit. If all goes according to plan, folks seeking multilib x86_64 will no longer have to resort to CLFS and put up with the associated irritation (recall that CLFS has different, more ambitious goals). In summary, this changes the current native build method into a hybrid native/cross build method whereby cross compilation is leveraged only for the initial "bootstrap" away from the host. After I've done a bit more testing, I'll put up some details on how it actually works with rationales etc. Stay tuned.. Regards Greg From diy-linux-dev@diy-linux.org Sat Oct 27 23:44:15 2007 From: diy-linux-dev@diy-linux.org (Alf mel) Date: Sat, 27 Oct 2007 17:44:15 -0600 Subject: Multilib build method In-Reply-To: <20071027064843.GA8402@eyo32.local> References: <20071027064843.GA8402@eyo32.local> Message-ID: <200710271744.15299.alf@mypals.org> On Saturday 27 October 2007 12:48:43 am Greg Schafer wrote: > But the best part of all - it enables *sane* x86_64 multilib!! And when I > say "sane" I mean understandable by mere mortals and with minimal > hackery. And it doesn't really matter that much how sane the host is, or > whether the host is 32 or 64 bit. Did I hear (read) you correctly? A sane way of building a multilib system? That is, with 32-bit and 64-bit binary support? Congrats to Alexander Patrakov and to our "benevolent dictator" Greg Schafer! I can't wait to play with the new toy! -- @ - Alf From diy-linux-dev@diy-linux.org Tue Oct 30 03:32:20 2007 From: diy-linux-dev@diy-linux.org (Greg Schafer) Date: Tue, 30 Oct 2007 13:32:20 +1100 Subject: Multilib build method In-Reply-To: <20071027064843.GA8402@eyo32.local> References: <20071027064843.GA8402@eyo32.local> Message-ID: <20071030023220.GA2003@eyo32.local> On Sat, Oct 27, 2007 at 04:48:43PM +1000, Greg Schafer wrote: > In summary, this changes the current native build method into a hybrid > native/cross build method whereby cross compilation is leveraged only for > the initial "bootstrap" away from the host. > > After I've done a bit more testing, I'll put up some details on how it > actually works with rationales etc. Stay tuned.. Ok guys, here's a first stab at the Temptools phase. It works quite well in the general case eg: targeting i686-pc-linux-gnu. However, multilib x86_64 still needs more work because of the Chroot phase. The main differences are Binutils-Pass1 and GCC-Pass1 now become "Cross-Binutils" and "Cross-GCC". And the Pass2's of GCC and Binutils swap position ie: the build order now looks like this: - Pass 1's of Bash, Make, Sed, etc as per current - Cross Binutils - Cross GCC - Kernel Headers - Glibc 32-bit (ONLY for x86_64 Multilib.. obviously) - Glibc - Adjust Toolchain - 64-bit kernel (ONLY if targeting x86_64 and coming from i686) REBOOT into 64-bit kernel - Native Binutils - Native GCC - Gawk, Coreutils, Bzip2, etc, etc as per current There are 2 new environment variables, "$TARGET" and "$BUILD_HOST", both of which should be self-explanatory. Because we want to use cross compilation even when targetting i686-pc-linux-gnu from i686-pc-linux-gnu, we need to "fake" the build host. But even when targeting x86_64 multilib from i686, we still need to fake the build host to build the 32-bit glibc. For example: Real Host $TARGET $BUILD_HOST i686-pc-linux-gnu i686-pc-linux-gnu i686-fake-linux-gnu i686-pc-linux-gnu x86_64-unknown-linux-gnu(M) i686-fake-linux-gnu x86_64-unknown-linux-gnu x86_64-unknown-linux-gnu x86_64-fake-linux-gnu x86_64-unknown-linux-gnu x86_64-unknown-linux-gnu(M) x86_64-fake-linux-gnu i686-pc-linux-gnu x86_64-unknown-linux-gnu i686-pc-linux-gnu (M) denotes multilib As can be seen, the only combo that doesn't require faking of the build host is pure x86_64 from i686. But I think I'll just mandate the faked build host for the sake of uniformity. It should be easy to automate the setting of a faked $BUILD_HOST by installing config.guess in $HOME as part of the environment setup and putting some logic in the bash startup files of the build user. Will sort this detail out later.. Ok, the key to any build method is understanding the toolchain search paths ie: includes, libs, startfiles, etc. The beauty of using cross compilation for the initial toolchain is that searching of the host dirs is virtually eliminated. However, cross compiling Glibc can be a very complex business. There are all sorts of configuration options and questions. Do I need Glibc headers? Do I need a sysroot? What about all these cryptic --host, --build and --target switches? Arghhh, it's all too hard! Luckily, GCC has a "minimal" mode of operation whereby access to the target's headers and libraries is not required. This is signified by the definition of "inhibit_libc" in the Makefiles. I first touched upon this aspect a few years ago back here: http://www.diy-linux.org/pipermail/diy-linux-dev/2005-July/000575.html Basically, this mini GCC is configured with --disable-shared --disable-threads --disable-everything-in-sight. A critical point to understand is that this mini GCC is *perfectly* capable of compiling the Glibc libs AND the Linux kernel, but it won't be able to *link* anything (keeping in mind that unlike most normal userland sources, Glibc and the kernel take care of their own linking). But please note, this is definitely *NOT* the correct way to build a modern true cross compiler targeting Glibc, because that entails using Glibc headers and --with-sysroot. For yet *another* view on some of the above issues, I encourage you to take a look at the newish Eglibc project at http://www.eglibc.org/ which is essentially Glibc customized for embedded, and in particular this doc: http://www.eglibc.org/cgi-bin/viewcvs.cgi/*checkout*/branches/eglibc-2_5/libc/EGLIBC.cross-building It's worth noting those guys (Codesourcery) are toolchain professionals. I don't necessarily agree that their methods are suitable for our use case, but they are certainly interesting nonetheless. Anyhoo, to recap - our goal is to build a minimal cross compiler that doesn't need access to Glibc headers and libs. This has the *MASSIVE* advantage of allowing us to have an x86_64 multilib GCC that can compile both -m32 and -m64 for bootstrapping purposes. One last (obvious) thing to consider, if we're running a 32-bit kernel and we're targeting x86_64, we MUST be running a 64-bit kernel by the time we go to compile the native binutils. This is easily achieved by using the first-up cross toolchain to build a 64-bit kernel and rebooting into it. So, that's pretty much it. Following are the build commands. No silly patches, no brutal hackery, no unnecessary switches and variables. Overall it seems pretty clean to me, but of course it's not finished yet. There's also some interesting comments below, just prior to the native binutils. Not sure yet whether I'll somehow integrate all this into the current Refbuild or write a separate doc. Comments? Questions? Regards Greg #----------------------------------------------------------- # cross-binutils #----------------------------------------------------------- mkdir ../binutils-build cd ../binutils-build ../binutils-2.18/configure \ --target=$TARGET \ --build=$BUILD_HOST \ --disable-werror echo "MAKEINFO = :" >> Makefile make make install #----------------------------------------------------------- # cross-gcc #----------------------------------------------------------- mkdir ../gcc-build cd ../gcc-build ../gcc-4.2.2/configure \ --target=$TARGET \ --host=$BUILD_HOST \ --enable-languages=c \ --disable-shared \ --disable-threads \ --disable-libmudflap \ --disable-libssp \ --disable-libgomp make make install -j1 ln -sv libgcc.a `$TARGET-gcc -print-libgcc-file-name | sed 's/libgcc/&_eh/'` if [ "$DIY_ARCH" = x86_64 ]; then ln -sv libgcc.a `$TARGET-gcc -m32 -print-libgcc-file-name | sed 's/libgcc/&_eh/'` fi #----------------------------------------------------------- # kernel-headers #----------------------------------------------------------- case "$DIY_ARCH" in i386) KARCH=i386 ;; ppc) KARCH=powerpc ;; x86_64) KARCH=x86_64 ;; esac make mrproper make headers_install INSTALL_HDR_PATH=temp ARCH=$KARCH cp -rv temp/include $TT_PFX #----------------------------------------------------------- # glibc-32-bit : ONLY for x86_64 Multilib!! #----------------------------------------------------------- mkdir ../glibc-build cd ../glibc-build CC="$TARGET-gcc -m32" \ CFLAGS="-O2 -march=i486 -pipe" \ ../glibc-2.6.1/configure \ --host=i686-pc-linux-gnu \ --build=$BUILD_HOST \ --disable-profile \ --enable-add-ons \ --with-headers=${TT_PFX}/include \ libc_cv_initfini_array=yes \ libc_cv_forced_unwind=yes \ libc_cv_c_cleanup=yes make make install -j1 #----------------------------------------------------------- # glibc #----------------------------------------------------------- [ "$DIY_ARCH" = i386 ] && MARCH="-march=i486" mkdir ../glibc-build cd ../glibc-build CFLAGS="-O2 $MARCH -pipe" \ ../glibc-2.6.1/configure \ --host=$TARGET \ --build=$BUILD_HOST \ --disable-profile \ --enable-add-ons \ --with-headers=${TT_PFX}/include \ libc_cv_initfini_array=yes \ libc_cv_forced_unwind=yes \ libc_cv_c_cleanup=yes [ "$DIY_ARCH" = x86_64 ] && echo "slibdir=$TT_PFX/lib64 libdir=$TT_PFX/lib64" > configparms make make install -j1 ************************************************************ Ok, this next part is key, and it's quite interesting. We are using our cross toolchain to build native tools! This is achieved by manipulating the search paths for includes and libs/startfiles. Also take special note of the "$DIR_SUFFIX" use. ************************************************************ #----------------------------------------------------------- # native-binutils #----------------------------------------------------------- [ "$DIY_ARCH" = x86_64 ] && DIR_SUFFIX=64 mkdir ../binutils-build cd ../binutils-build CC="$TARGET-gcc -isystem $TT_PFX/include -B $TT_PFX/lib${DIR_SUFFIX}/" \ AR="$TARGET-ar" \ RANLIB="$TARGET-ranlib" \ ../binutils-2.18/configure \ --with-lib-path=${TT_PFX}/lib${DIR_SUFFIX} echo "MAKEINFO = :" >> Makefile make make install make -C ld clean make -C ld LIB_PATH=/lib${DIR_SUFFIX}:/usr/lib${DIR_SUFFIX} cp -v ld/ld-new ${TT_PFX}/bin #----------------------------------------------------------- # native-gcc #----------------------------------------------------------- case "$DIY_ARCH" in i386) DL=/lib/ld-linux.so.2; DL_HEADER=i386/linux.h ;; ppc) DL=/lib/ld.so.1; DL_HEADER=rs6000/sysv4.h ;; x86_64) DL=/lib64/ld-linux-x86-64.so.2; DL_HEADER=i386/linux64.h; DIR_SUFFIX=64 ;; esac sed -i.bak \ "s@${DL}@${TT_PFX}&@" gcc/config/${DL_HEADER} echo " /* Remove /usr/include from end of include search path. */ #undef STANDARD_INCLUDE_DIR #define STANDARD_INCLUDE_DIR 0" >> gcc/config/${DL_HEADER} sed -i.bak \ -e 's,\./fixinc\.sh,-c true,' \ -e '/^LIBGCC2_DEBUG/d' gcc/Makefile.in mkdir ../gcc-build cd ../gcc-build CC="$TARGET-gcc -isystem $TT_PFX/include -B $TT_PFX/lib${DIR_SUFFIX}/" \ ../gcc-4.2.2/configure \ --with-local-prefix=${TT_PFX} \ --enable-shared \ --disable-multilib \ --disable-bootstrap \ --enable-languages=c,c++ \ --enable-clocale=gnu \ --enable-threads=posix \ --enable-__cxa_atexit \ --disable-libstdcxx-pch make LDFLAGS=${LDFLAGS} make install -j1 ln -sv gcc ${TT_PFX}/bin/cc From diy-linux-dev@diy-linux.org Tue Oct 30 05:15:48 2007 From: diy-linux-dev@diy-linux.org (Jeremy Huntwork) Date: Tue, 30 Oct 2007 00:15:48 -0400 Subject: Multilib build method In-Reply-To: <20071030023220.GA2003@eyo32.local> References: <20071027064843.GA8402@eyo32.local> <20071030023220.GA2003@eyo32.local> Message-ID: <4726AFF4.2050500@linuxfromscratch.org> Greg Schafer wrote: > Not sure yet whether I'll somehow integrate all this into the current > Refbuild or write a separate doc. Comments? Questions? First of all, nice to see that it's working. Without realizing that you were already investigating this (or Alexander, too, for that matter) about a month ago I was playing with a similar setup, but ran into some trouble after adjusting the toolchain and attempting to build natively, and put the idea on hold. (Possibly, I needed to adjust the order of binutils and gcc as you did.) My motivation for trying was the current LFS LiveCDs for x86 which have 32-bit userspace, but an optional 64-bit kernel ostensibly for CLFS use. I thought it would be nice to be able to use the 64-bit kernel for building x86_64 natively, but obviously the build would have to be altered due to missing 64-bit libraries. It's nice to know that I was on a similar path. :) Anyway, here's a question. IIRC, the cross-compiler gets installed as $TARGET-gcc (as well as the other build tools, linker, assembler, etc.) Do you just symlink to those after the cross-build of Glibc? What extra steps are added to the 'Adjust Toolchain' section? -- JH From diy-linux-dev@diy-linux.org Tue Oct 30 05:53:06 2007 From: diy-linux-dev@diy-linux.org (Greg Schafer) Date: Tue, 30 Oct 2007 15:53:06 +1100 Subject: Multilib build method In-Reply-To: <4726AFF4.2050500@linuxfromscratch.org> References: <20071027064843.GA8402@eyo32.local> <20071030023220.GA2003@eyo32.local> <4726AFF4.2050500@linuxfromscratch.org> Message-ID: <20071030045306.GA2955@eyo32.local> On Tue, Oct 30, 2007 at 12:15:48AM -0400, Jeremy Huntwork wrote: > Anyway, here's a question. IIRC, the cross-compiler gets installed as > $TARGET-gcc (as well as the other build tools, linker, assembler, etc.) > Do you just symlink to those after the cross-build of Glibc? What extra > steps are added to the 'Adjust Toolchain' section? ??? Not sure what you're getting at. There is no need to symlink anything. This is the kind of thing I was referring to when I said this method tends to simplify some aspects of the build. Ohhh fsck. I just realized I forgot to include the adjust toolchain section.. oops.. sorry about that. Regards Greg #----------------------------------------------------------- # adjust-toolchain #----------------------------------------------------------- case "$DIY_ARCH" in i386) DL=/lib/ld-linux.so.2 ;; ppc) DL=/lib/ld.so.1 ;; x86_64) DL=/lib64/ld-linux-x86-64.so.2 ;; esac $TARGET-gcc -dumpspecs | sed \ "s,${DL},${TT_PFX}&," > \ `$TARGET-gcc -print-search-dirs | head -n 1 | awk '{ print $2 }'`specs From diy-linux-dev@diy-linux.org Tue Oct 30 06:02:05 2007 From: diy-linux-dev@diy-linux.org (Jeremy Huntwork) Date: Tue, 30 Oct 2007 01:02:05 -0400 Subject: Multilib build method In-Reply-To: <20071030045306.GA2955@eyo32.local> References: <20071027064843.GA8402@eyo32.local> <20071030023220.GA2003@eyo32.local> <4726AFF4.2050500@linuxfromscratch.org> <20071030045306.GA2955@eyo32.local> Message-ID: <4726BACD.7050701@linuxfromscratch.org> Greg Schafer wrote: > $TARGET-gcc -dumpspecs | sed \ > "s,${DL},${TT_PFX}&," > \ > `$TARGET-gcc -print-search-dirs | head -n 1 | awk '{ print $2 }'`specs This is what I mean, you're applying the specs change to $TARGET-gcc. So what do you do for the next package's configure script, (binutils if you're not building the kernel) CC=$TARGET-gcc? Or do you use --host and --build parameters? -- JH From diy-linux-dev@diy-linux.org Tue Oct 30 06:07:46 2007 From: diy-linux-dev@diy-linux.org (Greg Schafer) Date: Tue, 30 Oct 2007 16:07:46 +1100 Subject: Multilib build method In-Reply-To: <4726BACD.7050701@linuxfromscratch.org> References: <20071027064843.GA8402@eyo32.local> <20071030023220.GA2003@eyo32.local> <4726AFF4.2050500@linuxfromscratch.org> <20071030045306.GA2955@eyo32.local> <4726BACD.7050701@linuxfromscratch.org> Message-ID: <20071030050746.GA5730@eyo32.local> On Tue, Oct 30, 2007 at 01:02:05AM -0400, Jeremy Huntwork wrote: > This is what I mean, you're applying the specs change to $TARGET-gcc. So > what do you do for the next package's configure script, (binutils if > you're not building the kernel) CC=$TARGET-gcc? Yes (it's there - you missed it?). > Or do you use --host and --build parameters? No. We're native at that point so no need. Regards Greg From diy-linux-dev@diy-linux.org Tue Oct 30 06:15:54 2007 From: diy-linux-dev@diy-linux.org (Jeremy Huntwork) Date: Tue, 30 Oct 2007 01:15:54 -0400 Subject: Multilib build method In-Reply-To: <20071030050746.GA5730@eyo32.local> References: <20071027064843.GA8402@eyo32.local> <20071030023220.GA2003@eyo32.local> <4726AFF4.2050500@linuxfromscratch.org> <20071030045306.GA2955@eyo32.local> <4726BACD.7050701@linuxfromscratch.org> <20071030050746.GA5730@eyo32.local> Message-ID: <4726BE0A.2070804@linuxfromscratch.org> Greg Schafer wrote: > On Tue, Oct 30, 2007 at 01:02:05AM -0400, Jeremy Huntwork wrote: > >> This is what I mean, you're applying the specs change to $TARGET-gcc. So >> what do you do for the next package's configure script, (binutils if >> you're not building the kernel) CC=$TARGET-gcc? > > Yes (it's there - you missed it?). Indeed. I replied (and trimmed) too quickly and didn't even notice that you included the actual commands. :-/ Looking through them all now... -- JH From diy-linux-dev@diy-linux.org Tue Oct 30 06:25:59 2007 From: diy-linux-dev@diy-linux.org (Alexander E. Patrakov) Date: Tue, 30 Oct 2007 10:25:59 +0500 Subject: Multilib build method In-Reply-To: <20071030023220.GA2003@eyo32.local> References: <20071027064843.GA8402@eyo32.local> <20071030023220.GA2003@eyo32.local> Message-ID: <4726C067.6020202@ums.usu.ru> Greg Schafer wrote: > - Pass 1's of Bash, Make, Sed, etc as per current > AKA "update the host" :) > - Cross Binutils > - Cross GCC > - Kernel Headers > OK. According to your research, the procedure works for x86, ppc and x86_64. However, previously you mentioned troubles compiling cross-gcc for ia64 without glibc headers. Do these considerations still apply? > - Glibc 32-bit (ONLY for x86_64 Multilib.. obviously) > - Glibc > Hm. While building a multilib system from the very beginning certainly works, I think that some low-priority research should be directed to the procedure of upgrading a pure64 system to multilib on the "as-needed" basis, by installing the 32-bit libraries. Is this possible? > - Adjust Toolchain > I see your mail where you include this step. > - 64-bit kernel (ONLY if targeting x86_64 and coming from i686) > REBOOT into 64-bit kernel > OK > - Native Binutils > - Native GCC > - Gawk, Coreutils, Bzip2, etc, etc as per current > Well, at this point a well-tested native build is taking place. > There are 2 new environment variables, "$TARGET" and "$BUILD_HOST", both of > which should be self-explanatory. Because we want to use cross compilation > even when targetting i686-pc-linux-gnu from i686-pc-linux-gnu, we need to > "fake" the build host. But even when targeting x86_64 multilib from i686, we > still need to fake the build host to build the 32-bit glibc. For example: > > Real Host $TARGET $BUILD_HOST > i686-pc-linux-gnu i686-pc-linux-gnu i686-fake-linux-gnu > i686-pc-linux-gnu x86_64-unknown-linux-gnu(M) i686-fake-linux-gnu > x86_64-unknown-linux-gnu x86_64-unknown-linux-gnu x86_64-fake-linux-gnu > x86_64-unknown-linux-gnu x86_64-unknown-linux-gnu(M) x86_64-fake-linux-gnu > i686-pc-linux-gnu x86_64-unknown-linux-gnu i686-pc-linux-gnu > (M) denotes multilib > Does downgrading a 64-bit system to a 32-bit one work (possibly with the "linux32" wrapper)? > As can be seen, the only combo that doesn't require faking of the build host > is pure x86_64 from i686. But I think I'll just mandate the faked build host > for the sake of uniformity. +1. > Basically, this mini GCC is configured with --disable-shared > --disable-threads --disable-everything-in-sight. A critical point to > understand is that this mini GCC is *perfectly* capable of compiling the > Glibc libs AND the Linux kernel, but it won't be able to *link* anything > Is the resulting Glibc still fully functional, given that libgcc.a has been compiled without access to glibc headers? > #----------------------------------------------------------- > # cross-gcc > #----------------------------------------------------------- > mkdir ../gcc-build > cd ../gcc-build > > ../gcc-4.2.2/configure \ > --target=$TARGET \ > --host=$BUILD_HOST \ > --enable-languages=c \ > --disable-shared \ > --disable-threads \ > --disable-libmudflap \ > --disable-libssp \ > --disable-libgomp > > make > make install -j1 > Any advantages over make all-gcc && make install-gcc? > ln -sv libgcc.a `$TARGET-gcc -print-libgcc-file-name | sed 's/libgcc/&_eh/'` > if [ "$DIY_ARCH" = x86_64 ]; then > ln -sv libgcc.a `$TARGET-gcc -m32 -print-libgcc-file-name | sed 's/libgcc/&_eh/'` > fi > OK. This works because you don't disable multilib above. > #----------------------------------------------------------- > # kernel-headers > #----------------------------------------------------------- > case "$DIY_ARCH" in > i386) KARCH=i386 ;; > ppc) KARCH=powerpc ;; > x86_64) KARCH=x86_64 ;; > esac > > make mrproper > make headers_install INSTALL_HDR_PATH=temp ARCH=$KARCH > cp -rv temp/include $TT_PFX > OK. > #----------------------------------------------------------- > # glibc-32-bit : ONLY for x86_64 Multilib!! > #----------------------------------------------------------- > mkdir ../glibc-build > cd ../glibc-build > > CC="$TARGET-gcc -m32" \ > CFLAGS="-O2 -march=i486 -pipe" \ > ../glibc-2.6.1/configure \ > --host=i686-pc-linux-gnu \ > --build=$BUILD_HOST \ > --disable-profile \ > --enable-add-ons \ > --with-headers=${TT_PFX}/include \ > libc_cv_initfini_array=yes \ > libc_cv_forced_unwind=yes \ > libc_cv_c_cleanup=yes > > make > make install -j1 > > #----------------------------------------------------------- > # glibc > #----------------------------------------------------------- > [ "$DIY_ARCH" = i386 ] && MARCH="-march=i486" > > mkdir ../glibc-build > cd ../glibc-build > > CFLAGS="-O2 $MARCH -pipe" \ > ../glibc-2.6.1/configure \ > --host=$TARGET \ > --build=$BUILD_HOST \ > --disable-profile \ > --enable-add-ons \ > --with-headers=${TT_PFX}/include \ > libc_cv_initfini_array=yes \ > libc_cv_forced_unwind=yes \ > libc_cv_c_cleanup=yes > > [ "$DIY_ARCH" = x86_64 ] && > echo "slibdir=$TT_PFX/lib64 > libdir=$TT_PFX/lib64" > configparms > > make > make install -j1 > Good, but this still needs some tweaks in order to make the commands PM-friendly. Here we rely on the fact that the 64-bit glibc will overwrite 32-bit headers, and some package managers don't like such file conflicts. > > ************************************************************ > Ok, this next part is key, and it's quite interesting. We are using our > cross toolchain to build native tools! This is achieved by manipulating the > search paths for includes and libs/startfiles. Also take special note of the > "$DIR_SUFFIX" use. > ************************************************************ > > #----------------------------------------------------------- > # native-binutils > #----------------------------------------------------------- > [ "$DIY_ARCH" = x86_64 ] && DIR_SUFFIX=64 > > mkdir ../binutils-build > cd ../binutils-build > > CC="$TARGET-gcc -isystem $TT_PFX/include -B $TT_PFX/lib${DIR_SUFFIX}/" \ > AR="$TARGET-ar" \ > RANLIB="$TARGET-ranlib" \ > ../binutils-2.18/configure \ > --with-lib-path=${TT_PFX}/lib${DIR_SUFFIX} > echo "MAKEINFO = :" >> Makefile > > make > make install > > make -C ld clean > make -C ld LIB_PATH=/lib${DIR_SUFFIX}:/usr/lib${DIR_SUFFIX} > cp -v ld/ld-new ${TT_PFX}/bin > Quite understandable. The -B switch is the key to find startfiles. > #----------------------------------------------------------- > # native-gcc > #----------------------------------------------------------- > > case "$DIY_ARCH" in > i386) DL=/lib/ld-linux.so.2; DL_HEADER=i386/linux.h ;; > ppc) DL=/lib/ld.so.1; DL_HEADER=rs6000/sysv4.h ;; > x86_64) DL=/lib64/ld-linux-x86-64.so.2; DL_HEADER=i386/linux64.h; DIR_SUFFIX=64 ;; > esac > > sed -i.bak \ > "s@${DL}@${TT_PFX}&@" gcc/config/${DL_HEADER} > > echo " > /* Remove /usr/include from end of include search path. */ > #undef STANDARD_INCLUDE_DIR > #define STANDARD_INCLUDE_DIR 0" >> gcc/config/${DL_HEADER} > > sed -i.bak \ > -e 's,\./fixinc\.sh,-c true,' \ > -e '/^LIBGCC2_DEBUG/d' gcc/Makefile.in > > mkdir ../gcc-build > cd ../gcc-build > > CC="$TARGET-gcc -isystem $TT_PFX/include -B $TT_PFX/lib${DIR_SUFFIX}/" \ > ../gcc-4.2.2/configure \ > --with-local-prefix=${TT_PFX} \ > --enable-shared \ > --disable-multilib \ > --disable-bootstrap \ > --enable-languages=c,c++ \ > --enable-clocale=gnu \ > --enable-threads=posix \ > --enable-__cxa_atexit \ > --disable-libstdcxx-pch > > make LDFLAGS=${LDFLAGS} > > make install -j1 > ln -sv gcc ${TT_PFX}/bin/cc > Yes, this is just a normal native gcc, that partially overwrites cross-gcc. Which files remain from our cross-tools at this point? -- Alexander E. Patrakov From diy-linux-dev@diy-linux.org Tue Oct 30 06:37:30 2007 From: diy-linux-dev@diy-linux.org (Alexander E. Patrakov) Date: Tue, 30 Oct 2007 10:37:30 +0500 Subject: Multilib build method In-Reply-To: <4726C067.6020202@ums.usu.ru> References: <20071027064843.GA8402@eyo32.local> <20071030023220.GA2003@eyo32.local> <4726C067.6020202@ums.usu.ru> Message-ID: <4726C31A.20501@ums.usu.ru> I wrote: > Good, but this still needs some tweaks in order to make the commands > PM-friendly. Here we rely on the fact that the 64-bit glibc will > overwrite 32-bit headers, and some package managers don't like such > file conflicts. Oops. This applies to the equivalent of LFS Chapter 6 only. But still, it is an open question how to do this. -- Alexander E. Patrakov From diy-linux-dev@diy-linux.org Tue Oct 30 07:16:20 2007 From: diy-linux-dev@diy-linux.org (Greg Schafer) Date: Tue, 30 Oct 2007 17:16:20 +1100 Subject: Multilib build method In-Reply-To: <4726C067.6020202@ums.usu.ru> References: <20071027064843.GA8402@eyo32.local> <20071030023220.GA2003@eyo32.local> <4726C067.6020202@ums.usu.ru> Message-ID: <20071030061620.GA6019@eyo32.local> On Tue, Oct 30, 2007 at 10:25:59AM +0500, Alexander E. Patrakov wrote: > > - Pass 1's of Bash, Make, Sed, etc as per current > > > AKA "update the host" :) Installing to /usr updates the host. This doesn't :-) > > - Cross Binutils > > - Cross GCC > > - Kernel Headers > > > OK. According to your research, the procedure works for x86, ppc and > x86_64. However, previously you mentioned troubles compiling cross-gcc > for ia64 without glibc headers. Do these considerations still apply? I haven't tried, but I would guess ia64 still won't work. BTW, AFAICT even the Eglibc method probably won't work for ia64. > > - Glibc 32-bit (ONLY for x86_64 Multilib.. obviously) > > - Glibc > > > Hm. While building a multilib system from the very beginning certainly > works, I think that some low-priority research should be directed to the > procedure of upgrading a pure64 system to multilib on the "as-needed" > basis, by installing the 32-bit libraries. Is this possible? I haven't really thought about it, and don't intend to because it sounds kinda messy and is not likely to work. > Does downgrading a 64-bit system to a 32-bit one work (possibly with the > "linux32" wrapper)? I knew someone would ask this :-) No, it's not designed for that at all because it cannot work. Nobody sane would want to travel down that path so I'm not interested. But I s'pose it's possible for someone to dream up some creative use of linux32.. > Is the resulting Glibc still fully functional, given that libgcc.a has > been compiled without access to glibc headers? Refer my cross compile research from 2 years ago. The answer should be "yes" but I haven't actually checked this time around because I don't think it matters given that the goal is to produce the chroot phase. But it's probably worth checking so I'll look into it. > Any advantages over make all-gcc && make install-gcc? None that I can see. But using the *-gcc targets is old-school IMHO. > Good, but this still needs some tweaks in order to make the commands > PM-friendly. Here we rely on the fact that the 64-bit glibc will > overwrite 32-bit headers, and some package managers don't like such file > conflicts. Indeed (for the chroot phase). I'm not yet sure how to handle this. There appears to be no way to install only the Glibc libs. Headers yes, but libs no. Ideas anyone? > Yes, this is just a normal native gcc, that partially overwrites > cross-gcc. Which files remain from our cross-tools at this point? Don't know, don't care. It shouldn't matter. Regards Greg From diy-linux-dev@diy-linux.org Tue Oct 30 07:25:57 2007 From: diy-linux-dev@diy-linux.org (Alexander E. Patrakov) Date: Tue, 30 Oct 2007 11:25:57 +0500 Subject: Multilib build method In-Reply-To: <20071030061620.GA6019@eyo32.local> References: <20071027064843.GA8402@eyo32.local> <20071030023220.GA2003@eyo32.local> <4726C067.6020202@ums.usu.ru> <20071030061620.GA6019@eyo32.local> Message-ID: <4726CE75.20400@ums.usu.ru> Greg Schafer wrote: >> Does downgrading a 64-bit system to a 32-bit one work (possibly with the >> "linux32" wrapper)? >> > > I knew someone would ask this :-) No, it's not designed for that at all > because it cannot work. Nobody sane would want to travel down that path so > I'm not interested. But I s'pose it's possible for someone to dream up some > creative use of linux32. > OK, I will check this someday. The intended use is for building both 32-bit and 64-bit LiveCDs without rebooting (and, preferrably, in parallel, because this way both cores of my CPU will be busy and still produce diffable logs). >> Good, but this still needs some tweaks in order to make the commands >> PM-friendly. Here we rely on the fact that the 64-bit glibc will >> overwrite 32-bit headers, and some package managers don't like such file >> conflicts. >> > > Indeed (for the chroot phase). I'm not yet sure how to handle this. There > appears to be no way to install only the Glibc libs. Headers yes, but libs > no. Ideas anyone? > Mandate the use of non-empty $PM, erase unneeded stuff, transfer to the root filesystem with tar? -- Alexander E. Patrakov From diy-linux-dev@diy-linux.org Tue Oct 30 07:46:30 2007 From: diy-linux-dev@diy-linux.org (Greg Schafer) Date: Tue, 30 Oct 2007 17:46:30 +1100 Subject: Multilib build method In-Reply-To: <4726CE75.20400@ums.usu.ru> References: <20071027064843.GA8402@eyo32.local> <20071030023220.GA2003@eyo32.local> <4726C067.6020202@ums.usu.ru> <20071030061620.GA6019@eyo32.local> <4726CE75.20400@ums.usu.ru> Message-ID: <20071030064630.GA6134@eyo32.local> On Tue, Oct 30, 2007 at 11:25:57AM +0500, Alexander E. Patrakov wrote: > OK, I will check this someday. The intended use is for building both > 32-bit and 64-bit LiveCDs without rebooting (and, preferrably, in > parallel, because this way both cores of my CPU will be busy and still > produce diffable logs). If starting from 64-bit kernel, just chroot into a 32-bit file system image (via linux32) and build the 32-bit CD from there. > Mandate the use of non-empty $PM, erase unneeded stuff, transfer to the > root filesystem with tar? Yes of course. But then I lose the dual purpose nature of the current build. Regards Greg From diy-linux-dev@diy-linux.org Tue Oct 30 07:49:36 2007 From: diy-linux-dev@diy-linux.org (Alexander E. Patrakov) Date: Tue, 30 Oct 2007 11:49:36 +0500 Subject: Multilib build method In-Reply-To: <20071030064630.GA6134@eyo32.local> References: <20071027064843.GA8402@eyo32.local> <20071030023220.GA2003@eyo32.local> <4726C067.6020202@ums.usu.ru> <20071030061620.GA6019@eyo32.local> <4726CE75.20400@ums.usu.ru> <20071030064630.GA6134@eyo32.local> Message-ID: <4726D400.9020609@ums.usu.ru> Greg Schafer wrote: > If starting from 64-bit kernel, just chroot into a 32-bit file system image > (via linux32) and build the 32-bit CD from there. > Yes, this works if I use "linux32 chroot". -- Alexander E. Patrakov From diy-linux-dev@diy-linux.org Tue Oct 30 15:17:52 2007 From: diy-linux-dev@diy-linux.org (Alf mel) Date: Tue, 30 Oct 2007 08:17:52 -0600 Subject: Multilib build method In-Reply-To: <4726AFF4.2050500@linuxfromscratch.org> References: <20071027064843.GA8402@eyo32.local> <20071030023220.GA2003@eyo32.local> <4726AFF4.2050500@linuxfromscratch.org> Message-ID: <200710300817.52228.alf@mypals.org> On Monday 29 October 2007 10:15:48 pm Jeremy Huntwork wrote: > My motivation for trying was the current LFS LiveCDs for x86 which have > 32-bit userspace, but an optional 64-bit kernel ostensibly for CLFS use. > I thought it would be nice to be able to use the 64-bit kernel for > building x86_64 natively, but obviously the build would have to be > altered due to missing 64-bit libraries. It's nice to know that I was on > a similar path. :) I don't keep track of LFS much anymore, so here goes YADQ (yet another dumb question). I like the idea of having a CD with 32-bit userspace and both a 32-bit and 64-bit kernel. Is that possible? Would it be possible to boot with the 64-bit kernel and build everything without having to reboot into 64-bit? -- @ - Alf From diy-linux-dev@diy-linux.org Tue Oct 30 15:26:01 2007 From: diy-linux-dev@diy-linux.org (Alexander E. Patrakov) Date: Tue, 30 Oct 2007 19:26:01 +0500 Subject: Multilib build method In-Reply-To: <200710300817.52228.alf@mypals.org> References: <20071027064843.GA8402@eyo32.local> <20071030023220.GA2003@eyo32.local> <4726AFF4.2050500@linuxfromscratch.org> <200710300817.52228.alf@mypals.org> Message-ID: <47273EF9.3000601@ums.usu.ru> Alf mel wrote: > I don't keep track of LFS much anymore, so here goes YADQ (yet another dumb > question). I like the idea of having a CD with 32-bit userspace and both a > 32-bit and 64-bit kernel. Is that possible? Yes, such CD is even available for download (http://kerrek.linuxfromscratch.org/pub/lfs-livecd/lfslivecd-x86-6.3-r2130.iso). 64-bit kernels were available on LFS LiveCDs since the 6.2-3 version or something like this. To boot a 64-bit kernel, type "linux64" at the boot loader prompt. > Would it be possible to boot > with the 64-bit kernel and build everything without having to reboot into > 64-bit? > Yes. Try this from my CD :) -- Alexander E. Patrakov From diy-linux-dev@diy-linux.org Tue Oct 30 20:11:47 2007 From: diy-linux-dev@diy-linux.org (Jeremy Huntwork) Date: Tue, 30 Oct 2007 13:11:47 -0600 Subject: Multilib build method In-Reply-To: <200710300817.52228.alf@mypals.org> References: <20071027064843.GA8402@eyo32.local> <20071030023220.GA2003@eyo32.local> <4726AFF4.2050500@linuxfromscratch.org> <200710300817.52228.alf@mypals.org> Message-ID: <20071030191147.GA23711@linuxfromscratch.org> (Sorry in advance for breaking the threading)] Greg, I did a test run of your build, everything went smoothly (although I skipped the actual multilib settings). There is one little niggle I hit that I'm not seeing compensated for in your commands. For the first pass of binutils, I would see 'i686-fake-linux-gnu-ar: No such file or directory', or some variant of the same. Easily solved by adding AR='ar' to the configure command. But I'm wondering, did you encounter this at all? Also, I'd like to start adapting some of this method for the JH branch of LFS. But I'd probably ignore multilib completely still, at least for now. And I wouldn't include any instructions for cross-compiling a kernel and booting into it, i.e., starting from a x86 kernel on x86_64 hardware. Any objections, concerns, suggestions? -- JH From diy-linux-dev@diy-linux.org Tue Oct 30 22:22:22 2007 From: diy-linux-dev@diy-linux.org (Greg Schafer) Date: Wed, 31 Oct 2007 08:22:22 +1100 Subject: Multilib build method In-Reply-To: <20071030191147.GA23711@linuxfromscratch.org> References: <20071027064843.GA8402@eyo32.local> <20071030023220.GA2003@eyo32.local> <4726AFF4.2050500@linuxfromscratch.org> <200710300817.52228.alf@mypals.org> <20071030191147.GA23711@linuxfromscratch.org> Message-ID: <20071030212222.GA13485@eyo32.local> On Tue, Oct 30, 2007 at 01:11:47PM -0600, Jeremy Huntwork wrote: > There is one little niggle I hit that I'm not seeing compensated for in > your commands. For the first pass of binutils, I would see > 'i686-fake-linux-gnu-ar: No such file or directory', or some variant of > the same. Easily solved by adding AR='ar' to the configure command. But > I'm wondering, did you encounter this at all? No. You probably used --host= instead of --build= for the cross binutils. To clarify, when binutils top-level moved to autoconf-2.5x, --build became the correct switch to use. BTW, this is just one of many things clfs gets wrong. These switches can be very confusing. I recommend reading (at least 3 times) the following for all the gorey detail: info '(autoconf)Hosts and Cross-Compilation' > Also, I'd like to start adapting some of this method for the JH branch > of LFS. But I'd probably ignore multilib completely still, at least for > now. And I wouldn't include any instructions for cross-compiling a > kernel and booting into it, i.e., starting from a x86 kernel on x86_64 > hardware. Any objections, concerns, suggestions? Um, not for me to tell you what to do. But this thing ain't finished yet so I wouldn't rely on it until these builds get compared against current builds, ICA verified etc. As per normal, just ensure the ethical thing is done and give credit where credit is due. You should now be getting an understanding of how much work goes into this stuff :-) Regards Greg From diy-linux-dev@diy-linux.org Tue Oct 30 23:54:05 2007 From: diy-linux-dev@diy-linux.org (Jeremy Huntwork) Date: Tue, 30 Oct 2007 18:54:05 -0400 Subject: Multilib build method In-Reply-To: <20071030212222.GA13485@eyo32.local> References: <20071027064843.GA8402@eyo32.local> <20071030023220.GA2003@eyo32.local> <4726AFF4.2050500@linuxfromscratch.org> <200710300817.52228.alf@mypals.org> <20071030191147.GA23711@linuxfromscratch.org> <20071030212222.GA13485@eyo32.local> Message-ID: <4727B60D.3070007@linuxfromscratch.org> Greg Schafer wrote: > No. You probably used --host= instead of --build= for the cross binutils. Yep, that's it. > To clarify, when binutils top-level moved to autoconf-2.5x, --build became > the correct switch to use. BTW, this is just one of many things clfs gets > wrong. These switches can be very confusing. I recommend reading (at least 3 > times) the following for all the gorey detail: > > info '(autoconf)Hosts and Cross-Compilation' Will do. > Um, not for me to tell you what to do. But this thing ain't finished yet so > I wouldn't rely on it until these builds get compared against current > builds, ICA verified etc. As per normal, just ensure the ethical thing is > done and give credit where credit is due. Of course. And of course, I'll be doing as much testing as I can with whatever I hope to use. -- JH