For anyone looking to learn a bit more about filesystems ext2 strikes a great balance of simplicity and real world practicality, making it good to learn from. Code here https://elixir.bootlin.com/linux/v3.9/source/fs/ext2
Seems to me like either exFat or FAT32 would be simpler? There's billions of devices using it with millions more sold every year, and it's supported by all operating systems in current use so it also has greater real world practicality.
Those filesystems also don't trust their input and check bounds correctly. Mounting a malicious ext* partition can trigger code execution so it's never auto mounted and shouldn't be used on removable drives. This is a really weird limitation that the other common file systems don't have
Haha - there are many graybeards who would call most of systemd kinda crazy (myself included). But let's not get derailed into the old systemd debate here :)
Not sure what system you've used, but I've never seen automounting fail to work on ext4 volumes.
That being said, basically every file system ever is vulnerable to the malicious image. Kernel drivers assume that the metadata is sane enough and mistakes are from simple bugs, not deliberate attacks.
I never said it would fail to work, I said it's a massive vulnerability.
And no, implementations of other filesystems like FAT32 and ISO 9660 consider it a security bug when a malicious image gains arbitrarily code execution. What you described is exclusively a Linux desktop problem
exFAT is pretty portable these days and not encumbered by annoying file and disk size limitations. Its the default for SD cards and has had native support in Macos and windows for 15 years, linux has had kernel drivers for 5 years and FUSE/3rd party kernel modules for longer. Unless you need to interface with an ancient system there's no reason to use FAT on portable drives anymore.
Your link discusses how to install fusefs-exfat, which is a separate package, which is what all Linux distributions did before the 2019 licensing - Android had a separate licensed kernel module, etc. I can’t speak to the current state of BSD but a cursory search yields a lot of discussion.
> Companies can integrate exFAT into a specific group of consumer devices, including cameras, camcorders, and digital photo frames for a flat fee. Mobile phones, PCs, and networks have a different volume pricing model.
> exFAT is the official file system of SDXC cards. Because of this, any device not supporting exFAT, such as the Nintendo 3DS, may not legally advertise itself as SDXC compatible, despite supporting SDXC cards as mass storage devices by formatting the card with FAT32 or a proprietary file system tied to the device in question.
> On 28 August 2019, Microsoft published the exFAT specification and released the patent to the Open Invention Network members. The Linux kernel introduced native exFAT support with the 5.4 release in November 2019.
So yes, this has been annoying for years (2006-2019, at a minimum) and left a very bad taste in my mouth as someone who has a few dozen SD cards that regularly move around various embedded devices that are expensive, rarely receive updates, and work just fine (e.g. 3d printers, digital frames, regular printers, local storage on security cameras, adapters for retro hardware, etc.). The Wikipedia article notes that the Nintendo 3DS - 75.94 million units sold during its 2011-2019 lifespan [1] - lacked exFAT support, so it’s not like this only affected low-end/cheap/uncommon devices from small companies.
Windows “helpfully” disabled all ways to format large SD cards as FAT32 so I end up having to rely on a third party tool.
Of course, whenever possible, I use a sane and journaled filesystem but it’s rare that I have a choice. :)
This. and the BSDs or Minix are not in the "Open Invention Network" and as far as i can see, the BSDs still do not have a kernel extfat implementation as i far as i just have checked on their websites
I had trouble booting the Windows 10 installer from an exFAT formatted usb drive. That was thee years ago. So back then I had to recompress a cab file to make it smaller than 4GB using Linux. It worked but I was scratching my head that this was an issue.
I guess this could work but all instructions I could find back then were saying to create a FAT32 file system. I switched to exFAT after finding some installation files being large than 4GB. But then the stupid thing would not boot. So I managed to re-compress the largest cab file and got it below 4GB.
Edit:
But I am not sure an NTFS formatted stick is actually UEFI-bootable.
> But I am not sure an NTFS formatted stick is actually UEFI-bootable.
Practically - no.
But you can make a FAT32 standard bootable thumb drive, without install.wim which is > 4GB, make a second partition on that drive and format it NTFS, place everything (including install.wim) there and launch it from there after WinPE would run from the first partition.
> launch it from there after WinPE would run from the first partition.
How would you launch it from the NTFS partition; is that an option while in WinPE or will Windows automatically look for install.wim on other partitions? (met the 4GB problem in FAT32 and split install.wim in two for a successful USB install drive, but it would be interesting to learn how to do this via NFTS).
Thanks!
After the setup launches it would ask you for the language selection. After that you would be presented with a window which would allows you to install Windows or launch the 'Repair computer' wizard. Now just press Shift+F10, navigate to the other partition and launch setup.exe manually.
If you need, use diskpart to list available volumes:
diskpart
list volume
If the second, NTFS volume somehow don't receive a letter (and hence you can't navigate to it):
diskpart
list vol (note the number of the volume)
sel vol ## (the number of the value)
assign letter=K (or any other to your liking)
I make the setup USB from scratch using built-in Windows tools.
Using recent NT6 admin CMD, enter DISKPART, select your proper (USB) disk, and use the CLEAN command to prepare the USB drive. Safely eject the USB and reboot.
Reinsert the USB and run Diskpart again, select your proper disk, and enter CONVERT MBR to prepare it for traditional widespread bootability. This writes a Master Boot Record up to the first 440 bytes of sector 0, leaving the small remainder of sector 0 blank to allow room for a primary partition table.
Enter CREATE PARTITION PRIMARY to make a single blank partition encompassing the entire drive.
Enter ACTIVE to make the selected primary partition bootable, or it's going to do no such thing for you in BIOS.
Enter FORMAT FS=FAT32 LABEL="yourLabel" QUICK which will quick-format the partition like any other FAT32 volume.[0] Fat32 was originally the minimal default filesystem intended for UEFI firmware to boot from. But NTFS containing a proper EFI folder now usually also works just as well with lots of firmware, so alternatively FORMAT FS=NTFS LABEL="yourLabel" QUICK is mostly OK too if there is a need for storing any single file which would be too big for FAT32. Either way will create a (hidden) original NT6 bootsector for that volume, either a FAT32-NT6 or a NTFS-NT6 bootsector respectively.
(Test-booting in BIOS to USB at this point - without any boot files on the volume yet - should show "An operating system was not found. Try disconnecting any drives that don't contain an operating system. Press any key to restart", which is a sign of potential progress. Otherwise less-generic approaches may be needed before going forward. Test booting by UEFI will not be conclusive yet.)
While in Diskpart your new (hopefully labeled) volume will now have an alphabetical "drive" letter automatically assigned and you should now be able to view its contents using the mouse in File Explorer, there should only be folders for SystemVolumeInformation and RecycleBin, if that. (Sometimes you may need to assign a volume letter yourself, which is easy in Diskpart or the GUI Disk Manager.)
Exit diskpart and you will be back at admin CMD line, which window can then be closed.
In File Explorer, copy the boot files (bootmgrs, BOOT folder, and EFI folder) from the mounted Windows Setup ISO, over to the target formatted USB volume.
(Test-booting in BIOS to USB at this point - with only boot files on the volume so far - will show "Windows Boot Manager / Windows failed to start . . . ENTER=OS Selection", and hitting Enter takes you to the Windows Setup bootmenu entry which is a sign of further progress. Test booting using UEFI shows "Windows Boot Manager / Windows failed to start . . . ENTER=OS Selection / ESC=UEFI Firmware Settings", also progressing to the Windows Setup bootentry. Otherwise, less-generic approaches may be needed before going forward.)
In File Explorer, copy the remaining folders from the mounted Windows Setup ISO, over to the target USB volume. That will end up with the entire contents of the mounted ISO finally present on the USB. But not the least bit dependent on the DVD boot process any more.
Full successful booting should now be possible, BIOS or UEFI on any truly decent mainboard. Analogous to how the physical setup DVD works.
Much simpler & quicker to prepare than it sounds. The most time-consuming thing as usual is waiting for the large install.wim file to be copied from the mounted ISO to the target USB.
For BIOS booting, the mainboard will read the MBR/partitionTable (sector 0) of the designated boot drive which will then direct the boot routine to the active partition's bootsector on that drive, which will then proceed to read the bootsector of that partition, which then reads the bootsector's designated boot files (very commonly) located on the root of that same partition. For BIOS these files are Bootmgr(s), the BOOT folder, and Bootnxt if present. Just like it always was for NT6 before UEFI.
Now also on the Windows Startup media for more versatility there is an EFI folder present so the drive can boot using UEFI without the mainboard needing to utilize CSM/BIOS. UEFI doesn't depend on a coherent MBR, "active" partition, volume bootsector, nor bootmgr (and its associates) in the root of the volume. Just finds a nice accessible correct EFI folder on a firmware-readable filesystem supported by UEFI on that particular mainboard firmware. The ideal mainboard can use UEFI to boot from MBR or GPT layout drives, having a valid EFI folder on either a FAT32 or NTFS volume. A defect sometimes appears when the USB drive needs to be layed out GPT-style to boot just because the mainboard UEFI firmware is garbage.
Oh, well, GIGO.
.
[0] For "yourLabel" I like to use a descriptive abbreviation positively identifying both the drive and the partitionNumber on the drive, as a label for each volume. While keeping the volume label to 11 characters or less as nature intended. Like BLU8GB_P1 for the first (usually only) fat32 partition on a blue 8GB USB drive. Or BLU8GB_N1 if it was NTFS. Another ok labeling option is something like "W11_23H2v2" which gives you an idea of what is on the drive, but instead I still prefer to use the volume label to pinpoint the hardware, then put a more lengthy text file in the root on the volume, named appropriately to document what the software version is.
Unlabeled volumes autoidentify like "Local Drive (X:)" or "Removable Drive (Y:)" so don't let that fool you into thinking they are already labeled. I would take action and label them so they are less of an accident waiting to happen. Labeling can even be accomplished in the File Explorer GUI. Any time in the future that you plug in an unlabeled drive, you want it to contain the only unlabeled volume(s) you see.
I remember even into the early 2010s we would still just make for example /boot ext2 for simplicity sake (perhaps?). Is that still in practice? Haven’t been in the OS deployment weeds for a minute..
And eons ago, people made fun of me for using MS-DOS and loadlin as a bootloader for Linux. (This always worked fine for me, and MS-DOS was simple to make boot and install tools on with hardware of the time. A functional-enough MS-DOS installation took up a trivial amount of space on a garden-variety PC of the day, and it was easy to pare down.)
Fast forward 20 or 30 years: We're back to using a FAT[32] partition for booting, just with a very different mini-OS to do the job.
I dunno that it’s really all that different. EFI is sort of what DOS would likely have become if MS had kept with it. It uses PE format, has back slash paths, has the same .3 extensions, and so on. There are certainly many similarities.
What's different is that using loadlin on MS-DOS was "lol, fuckin' newb" material (even though it was exquisitely functional and ridiculously easy to use), while the potentially very similar EFI system is a heralded as a champion.
I guess I am trying to express my annoyance with the Linux crowd's fickle nature over the years.
My own (perhaps old) hardware gives me a choice of old-school BIOS, or more-modern UEFI.
(But perhaps amusingly-counter to my original suggestion, old-ass archaic emulated BIOS seems to still be preferred in some spaces in the VM-using crowd -- at least where things like libvirt are involved. In those spaces, BIOS usually works fine and it generally boots faster than *EFI.)
I use ext4 (for /boot) but I always enable "sync". I always imagine that makes it less likely to suffer corruption but I have no actual evidence for that. I might be making it up.
By any chance does that include interoperability with the ext2 implementation used on theater projectors? Not a gotcha, if that’s been solved that would make my life tremendously easier.
As far as I know literally all you have to do is mount with -t ext4 (which presumably will just happen by default for ext2 superblocks after this.) As far as I know, it will definitely not automatically enable ext2/ext3-incompatible features, you have to do this manually (though doing so is relatively easy.)
the ext4 driver doesn't (in fact, cannot) upgrade file systems missing features. If you mount an ext2 file system from the 1990s with the ext4 driver, it'll stay at the old feature set, for better and worse.
You can use tune2fs to upgrade file systems, downgrading is not (usually) possible without a full backup and restore.
I'm not very familiar with Linux filesystem drivers, but it seems odd that there's separate ext2, presumably ext3, and ext4 drivers when the latter are (100%?) backwards-compatible with the former filesystems.
That happened because the code changes from ext2 to ext3, and later from ext3 to ext4, were very invasive, and filesystems is one place where stability and reliability is very important. So instead of risking introducing data-loss bugs, they forked the code. It was always the plan that the older code would be removed some time later, once the newer code had proven its stability for a while.
The on-disk format for these three is the same; the difference is that the later ones understand more of the "incompatible" additions to the on-disk format (IIRC, the main ones being the journal for ext3, and extents for ext4, but there are others). You can convert an "ext2" filesystem into an "ext4" one by just enabling the relevant filesystem format flags on the superblock, and making a few necessary adjustments (for instance, creating a journal when enabling the "has journal" flag).
IIRC ext3 was folded into the ext4 driver. If I was guessing, I would say that they preserved ext2 separately this long because it's a much simpler chunk of code to maintain. (And it's simple, known, stable code, while ext4 continues to undergo active feature development. Not sure how likely that angle is.)
I really appreciate all this slow but inevitable work to solving the 2038 problem. In the wake of Y2K when we first talked about 2038 it was a joke, "ah we'll all be retired then and anyway computers will be totally different". But here we are now, 24 years later and 14 years before the actual deadline. Thankfully folks are doing the work of fixing the deeper system limitations.
I'm retired now but fully expecting to dust off a bunch of Unix skills in late 2037, early 2038. I suspect by then many people who know how to, say, build a Linux kernel from the command line will either be dead, retired, or too busy doing more fun things.
I don't understand the immediate concern about using 32 bits for file creation and modification times. Its reasonable to assume that if the value has overflown and is very low that its not indicating a date from 1970. There is probably room to add a single bit indicating if the epoch is 1970 or 2038.
This is completely orthogonal, and not particularly relevant. That said, I didn't know this was the case. In fact, I'd assume an unspecified mkfs would complain these days.
1) mkfs is a userspace utility. Its defaults have no consideration for which kernel driver implements this interface; I'd bet a dollar that mkfs has been hitting the ext4 driver for ages.
2) As stated in sister comments, support for actual ext2 filesystems is unaffected.
3) Using a very simple fs by default seems like a good thing. FAT (variants) has survived forever by being dirt-simple, despite its faults.
4) Who in the world doesn't specify options to mkfs? It's one of the few operations that really doesn't have "sane defaults".
Is there a "better" default fs? I dunno. I've looked at a few, but most seem... troubled. So many trade-offs.
