[Samba] (DeviceIoControl, FSCTL_SET_SPARSE)

[brian lamb via samba]

Im having an issue with what I believe might be pinned down to a need for a
proper samba configuration.



The problem: sparse writes (not sure which, either write sparse files to a
system that doesn't support them, or vice versa)



The system:

Zfs-on-linux, samba share



Accessing system which is erroring:

Win7x64_ult, freefilesync



Error message



NOTHING I have done to it is fixing, including

strict allocate = yes

strict allocate = no

share:fake_fscaps = 0

#share:fake_fscaps = 0



Current samba config:

[global]

#security = user

#passdb backend = tdbsam



workgroup = MYGROUP

#strict allocate = yes



#kill printing

printing = bsd

printcap name = /dev/null

load printers = no

#spoolss

disable spoolss = yes



#aim to fix windows freefilesync copy symlinks not working

#follow symlinks = yes

#wide links = yes



#another zpool implemention style testing with, didnt help

#usershare path = /san/vault

#max shares = 0 is disabled

#usershare max shares = 100

#usershare allow guest = no

#share:fake_fscaps = 0



[homes]

comment = Home Directories

valid users = %S, %D%w%S

browseable = No

read only = No

inherit acls = Yes



[san]

comment = ZFS 'san' pool Backup File Server

path = /san/vault

#share:fake_fscaps = 0

#ironically doesnt make it readonly, however, worked in favor of
writeabe, valid users, and browseable yes, on arch

#read only = yes

write list = @sys1



Writing is done on windows machine to this path, non mapped

//hostname/san/backup/..



Reference urls

http://www.freefilesync.org/forum/viewtopic.php?f=2
<http://www.freefilesync.org/forum/viewtopic.php?f=2&t=1650&p=13722#p13722>
&t=1650&p=13722#p13722

https://github.com/zfsonlinux/zfs



same problem, now on rhel7.3, followed me from an archx64 system, same pool
https://aur.archlinux.org/packages/zfs-linux-git/



Note, I tried last night to create instead a sparse vol, and mount it after
mkfs an ext3 filesystem on there, I didn't map to it separately using samba,
but I did a test write to that dir, and still, same problem. I didn't know
how feasible it was to actually think that a sparse vol, would solve a
sparse file write issue, but it was worth a try to me.



Any ideas, ideals a proper samba config that will fix this in one shot, are
warmly welcomes, I have TB's of data hanging on a thread, getting a bit
scattered, and causing a lot of other drives to have to suffer high head
writes because of it. One failure and im on a downhill slide. Time is of
the essence. I have already persued a ZFS driver issue / bug path, a
FreefileSync issue / bug path, and now im out of options I think, unless its
just the way I mounted or created my pool, OR what I suspected a bad
Property on ZFS, or the samba config.



Regards,

Brian

[David Disseldorp via samba]

Hi Brian,

Post by brian lamb via samba
Im having an issue with what I believe might be pinned down to a need for a
proper samba configuration.

Which version of Samba are you using?

Post by brian lamb via samba
The problem: sparse writes (not sure which, either write sparse files to a
system that doesn't support them, or vice versa)

What doesn't support sparse writes? How do you know that the problem is
related to sparse files?

Post by brian lamb via samba
Zfs-on-linux, samba share

ZFS on Linux + Samba supports sparse files...

Post by brian lamb via samba
Win7x64_ult, freefilesync
Error message

What's the error message? Has this application ever worked successfully
against Samba, or only against a Windows share?

Post by brian lamb via samba
NOTHING I have done to it is fixing, including
strict allocate = yes
strict allocate = no
share:fake_fscaps = 0
#share:fake_fscaps = 0

Windows clients request sparse file semantics explicitly, by calling
FSCTL_SET_SPARSE. Samba then retains this flag in the "user.DOSATTRIB"
extended attribute for the given file.
One way to modify this behaviour would be to set "store dos attributes"
to "no" in smb.conf - this would see Samba respond to any
FSCTL_SET_SPARSE request with NT_STATUS_INVALID_DEVICE_REQUEST. That
said, I have no idea whether or not this will fix the application error
that you're seeing.

Cheers, David

[brian lamb via samba]

-----Original Message-----
Subject: Re: [Samba] (DeviceIoControl, FSCTL_SET_SPARSE)

I wanted to also mention, that during this sparse write, I have captured the strace, which shows the file attributes specifically that it is trying to pass.

I see a difference, for e.g. I see that its st_size is 0 for the file that’s causing the failure, which this is a tmp sparse file so the application can add a placeholder for the file, so that it can do a fail safe copy I presume. (also turning off the fail safe file copy doesn’t help).

Strace below for one of several PID's that were running. Not sure if the results will be the same on all pids, or why I even see multiple PID's for smb when I run ps -ax. Note, the trace I copied the portion that goes all the way from the previous file's filename portion, to the following filename, so that you can place yourself in the log. Also note, the first file copied in successfully. Also note, the program is designed now to delete file if found, then add new. Also note, the file does exist, and if the other files exist, it only complains when they are sparse, and ONLY these .tmp files are the sparse files, not sure really why they are there to be honest with you, if fail-safe copy mode is off....
------------------------
newfstatat(8, "MSIAfterburnerSetup.zip", {st_mode=S_IFREG|0744, st_size=40376862, ...}, 0) = 0
fcntl(21, F_SETLKW, {type=F_RDLCK, whence=SEEK_SET, start=35036, len=1}) = 0
fcntl(21, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=35036, len=1}) = 0
newfstatat(8, "MPC-HCPortable_1.7.10.paf.exe", {st_mode=S_IFREG|0744, st_size=12670432, ...}, 0) = 0
fcntl(21, F_SETLKW, {type=F_RDLCK, whence=SEEK_SET, start=38448, len=1}) = 0
fcntl(21, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=38448, len=1}) = 0
newfstatat(8, "esetsmartinstaller_enu.exe", {st_mode=S_IFREG|0744, st_size=2870984, ...}, 0) = 0
fcntl(21, F_SETLKW, {type=F_RDLCK, whence=SEEK_SET, start=21484, len=1}) = 0
fcntl(21, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=21484, len=1}) = 0
newfstatat(8, "My Family Tree.zip.ffs_tmp", {st_mode=S_IFREG|0544, st_size=0, ...}, 0) = 0
fcntl(21, F_SETLKW, {type=F_RDLCK, whence=SEEK_SET, start=11552, len=1}) = 0
fcntl(21, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=11552, len=1}) = 0
newfstatat(8, "2006-2008", {st_mode=S_IFDIR|0755, st_size=9, ...}, 0) = 0
fcntl(21, F_SETLKW, {type=F_RDLCK, whence=SEEK_SET, start=6300, len=1}) = 0
fcntl(21, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=6300, len=1}) = 0
newfstatat(8, "kodi-16.1-Jarvis.exe", {st_mode=S_IFREG|0744, st_size=87266194, ...}, 0) = 0
fcntl(21, F_SETLKW, {type=F_RDLCK, whence=SEEK_SET, start=34432, len=1}) = 0
fcntl(21, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=34432, len=1}) = 0
newfstatat(8, "expressBuilder.tar.gz", {st_mode=S_IFREG|0744, st_size=1055704, ...}, 0) = 0

[brian lamb via samba]

To whomever reads this,
This post I accidentally sent to david himself, I am now sending it to the group. It will be out of order, and includes the strace.

Regards
Brian

My apologies for any inconvenience.

--------------


Hi David,

Thanks for your response.

The Error message is for e.g.:

Cannot write file attributes of "\\MYHOST\backup\sparse\My Family Tree 6.0.zip.ffs_tmp".
Error Code 1: Incorrect function. (DeviceIoControl, FSCTL_SET_SPARSE)

Samba Version:
Samba version 4.4.4
PID Username Group Machine Protocol Version Encryption Signing
----------------------------------------------------------------------------------------------------------------------------------------
21582 backupadmin sys1 myhost (ipv4:192.168.1.99:55068) SMB2_10 - -

I just tried the option " store dos attributes = no"
Im getting the same error as above still unfortunately, but im doing some more thorough testing on that. WOud you happen to know if I want the other two options set, and to what, at all? (> strict allocate & share:fake_fscaps = 0)

The application (freefilesync) has worked successfully against Samba before I rebuilt the array to a raidz2, from a zfs linear span (no raid parity disk, still other features worked, scrub, crc checking, etc..) and I had no problem in the 6TB 3 disk span. Now I added three disks, , its now ~8TB with two parity disks. The only thing I can imagine different is the freefilesync program, since there was a gap in time when I switched out the raid array, it didn’t work from the get go(the sparse part, same errors) but this was about a month or two, and a new version of freefilesync I believe too. I noted tht freefilesync has supported sparse for a long time, and I posted the url for the freefilesync forum, where the author of the app himself suspected it was possibly samba, see that post.

Much Appreciated David!

Regards Brian

-----Original Message-----
From: samba [mailto:samba-***@lists.samba.org] On Behalf Of David Disseldorp via samba
Sent: Tuesday, February 14, 2017 2:55 AM
To: brian lamb via samba <***@lists.samba.org>
Subject: Re: [Samba] (DeviceIoControl, FSCTL_SET_SPARSE)

Hi Brian,

Post by brian lamb via samba
Im having an issue with what I believe might be pinned down to a need
for a proper samba configuration.

Which version of Samba are you using?

Post by brian lamb via samba
The problem: sparse writes (not sure which, either write sparse files
to a system that doesn't support them, or vice versa)

What doesn't support sparse writes? How do you know that the problem is related to sparse files?

Post by brian lamb via samba
Zfs-on-linux, samba share

ZFS on Linux + Samba supports sparse files...

Post by brian lamb via samba
Win7x64_ult, freefilesync
Error message

What's the error message? Has this application ever worked successfully against Samba, or only against a Windows share?

Post by brian lamb via samba
NOTHING I have done to it is fixing, including
strict allocate = yes
strict allocate = no
share:fake_fscaps = 0
#share:fake_fscaps = 0

Windows clients request sparse file semantics explicitly, by calling FSCTL_SET_SPARSE. Samba then retains this flag in the "user.DOSATTRIB"
extended attribute for the given file.
One way to modify this behaviour would be to set "store dos attributes"
to "no" in smb.conf - this would see Samba respond to any FSCTL_SET_SPARSE request with NT_STATUS_INVALID_DEVICE_REQUEST. That said, I have no idea whether or not this will fix the application error that you're seeing.

Cheers, David

--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba

[David Disseldorp via samba]

Post by brian lamb via samba
Thanks for your response.
Cannot write file attributes of "\\MYHOST\backup\sparse\My Family Tree 6.0.zip.ffs_tmp".
Error Code 1: Incorrect function. (DeviceIoControl, FSCTL_SET_SPARSE)

Hmm, that seems to indicate that the initial fsctl request is failing.
Could you please try manually setting a file on the Samba share as
sparse with "fsutil sparse setflag <file_on_samba_share>":
https://technet.microsoft.com/en-us/library/cc788025(v=ws.11).aspx

Does it succeed when Samba is configured as per below?

Post by brian lamb via samba
Samba version 4.4.4
PID Username Group Machine Protocol Version Encryption Signing
----------------------------------------------------------------------------------------------------------------------------------------
21582 backupadmin sys1 myhost (ipv4:192.168.1.99:55068) SMB2_10 - -
I just tried the option " store dos attributes = no"
Im getting the same error as above still unfortunately, but im doing some more thorough testing on that. WOud you happen to know if I want the other two options set, and to what, at all? (> strict allocate & share:fake_fscaps = 0)

Given that the application is complaining about FSCTL_SET_SPARSE
failure, I would guess that it requires sparse-file support. Please
set "store dos attributes = yes" and drop the fake_fscaps setting.

Cheers, David

[brian lamb via samba]

Hi David,

Configured as you mention, and prior with the other tweaks I had in the first place, Im getting the same results, which is "FSUTIL utility requires a Local NTFS Volume", and mapping it.

I then went to contig, `contig -n X:\share\sparesefile 0`, as well as a few other size. Maybe im using it wrong, but some testing on contig has only "Access is denied" when I try to write to anything.

Oh one change, I put these settings only in Global (only, as opposed to also in the pool definition), I just realized. I believe global should be enough shouldn’t it?

[global]
# ZFS SPARSE
#share:fake_fscaps = 0
store dos attributes = yes
strict allocate = yes

I hadn’t heard any mention of the importance of strict allocate, would that setting be wise then?

What are some other tools I might try that with? I can try straight on the linux too, just as a test, if there is any tools recommended to test that with. I will also look something up for that tomorrow as well.

Thanks for the info, much appreciated.
-Brian

-----Original Message-----
From: samba [mailto:samba-***@lists.samba.org] On Behalf Of David Disseldorp via samba
Sent: Wednesday, February 15, 2017 5:23 AM
To: brian lamb via samba <***@lists.samba.org>
Subject: Re: [Samba] (DeviceIoControl, FSCTL_SET_SPARSE)

Post by brian lamb via samba
Thanks for your response.
Cannot write file attributes of "\\MYHOST\backup\sparse\My Family Tree 6.0.zip.ffs_tmp".
Error Code 1: Incorrect function. (DeviceIoControl, FSCTL_SET_SPARSE)

Hmm, that seems to indicate that the initial fsctl request is failing.
Could you please try manually setting a file on the Samba share as sparse with "fsutil sparse setflag <file_on_samba_share>":
https://technet.microsoft.com/en-us/library/cc788025(v=ws.11).aspx

Does it succeed when Samba is configured as per below?

Post by brian lamb via samba
Samba version 4.4.4
PID Username Group Machine Protocol Version Encryption Signing
----------------------------------------------------------------------------------------------------------------------------------------
21582 backupadmin sys1 myhost (ipv4:192.168.1.99:55068) SMB2_10 - -
I just tried the option " store dos attributes = no"
Im getting the same error as above still unfortunately, but im doing
some more thorough testing on that. WOud you happen to know if I want
the other two options set, and to what, at all? (> strict allocate &
share:fake_fscaps = 0)

Given that the application is complaining about FSCTL_SET_SPARSE failure, I would guess that it requires sparse-file support. Please set "store dos attributes = yes" and drop the fake_fscaps setting.

Cheers, David

--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba

[Rowland Penny via samba]

On Thu, 16 Feb 2017 01:20:46 -0800

Post by brian lamb via samba
Hi David,
Configured as you mention, and prior with the other tweaks I had in
the first place, Im getting the same results, which is "FSUTIL
utility requires a Local NTFS Volume", and mapping it.

On the ZFS machine, are you using 'vfs_zfsacl' ?

On the ext3 (why ext3 not ext4 ?) have you read this:

https://wiki.samba.org/index.php/File_System_Support#ext3

and are you using 'vfs_acl_xattr' ?

Rowland

[brian lamb via samba]

Hi Rowland,

I see this in the ZFS props, if that’s related, you might be referring to something in ext3 that im not aware of at the moment.

pool/myshare aclinherit restricted default
pool/myshare acltype off default

So, if you heard that I was going to use ext3, that was only as a solution, and that over ext4 due personal preference, since I made a mistake once on a reboot during write and suffered bad data loss. I gave up on ext4, in favor of ext3 shortly after, and have since, for years. However, I wanted to mention one thing, and you may have shed light on part of the issue. There is no Format other than zfs create, no ext3, I have not made a filesystem on it. I just went ahead and straight wrote to it. I have a post on stack overflow asking this exact question, if indeed I really need to format a zfs, but I'm not getting concrete answers yet I think.

Let me know if that helped answer, or else I can do some more digging around for you on that.

Thanks much.

Brian


-----Original Message-----
From: samba [mailto:samba-***@lists.samba.org] On Behalf Of Rowland Penny via samba
Sent: Thursday, February 16, 2017 1:38 AM
To: ***@lists.samba.org
Subject: Re: [Samba] (DeviceIoControl, FSCTL_SET_SPARSE)

On Thu, 16 Feb 2017 01:20:46 -0800

Post by brian lamb via samba
Hi David,
Configured as you mention, and prior with the other tweaks I had in
the first place, Im getting the same results, which is "FSUTIL
utility requires a Local NTFS Volume", and mapping it.

On the ZFS machine, are you using 'vfs_zfsacl' ?

On the ext3 (why ext3 not ext4 ?) have you read this:

https://wiki.samba.org/index.php/File_System_Support#ext3

and are you using 'vfs_acl_xattr' ?

Rowland

--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba

[Rowland Penny via samba]

On Thu, 16 Feb 2017 01:54:24 -0800

Post by brian lamb via samba
Hi Rowland,
I see this in the ZFS props, if that’s related, you might be
referring to something in ext3 that im not aware of at the moment.
pool/myshare aclinherit
restricted default
pool/myshare acltype
off default

Samba provides a link between Samba and ZFS, see here for the manpage:

https://fossies.org/linux/samba/docs/manpages/vfs_zfsacl.8

Post by brian lamb via samba
So, if you heard that I was going to use ext3, that was only as a
solution, and that over ext4 due personal preference, since I made a
mistake once on a reboot during write and suffered bad data loss. I
gave up on ext4, in favor of ext3 shortly after, and have since, for
years.

ext4 is basically ext3 with updates and some of the mount options built
in, 'acl' being one of these.

Post by brian lamb via samba
However, I wanted to mention one thing, and you may have shed
light on part of the issue. There is no Format other than zfs create,
no ext3, I have not made a filesystem on it. I just went ahead and
straight wrote to it. I have a post on stack overflow asking this
exact question, if indeed I really need to format a zfs, but I'm not
getting concrete answers yet I think.

I have no idea, I do not use ZFS, mainly because you cannot create a
Samba AD DC on it.

Post by brian lamb via samba
Let me know if that helped answer, or else I can do some more digging
around for you on that.

I was just trying to make a suggestion of something to try.

Rowland

[Klaus Hartnegg via samba]

Post by Rowland Penny via samba
On Thu, 16 Feb 2017 01:54:24 -0800

Post by brian lamb via samba
Hi Rowland,
I see this in the ZFS props, if that’s related, you might be
referring to something in ext3 that im not aware of at the moment.
pool/myshare aclinherit restricted default
pool/myshare acltype off default

https://fossies.org/linux/samba/docs/manpages/vfs_zfsacl.8

I am running Samba on ZFS without vfs_zfsacl, but with these zfs options:
- xattr=sa
- acltype=posixacl
- aclinherit=restricted

Samba is currenently 3.6.23-33.el6 from Scientific Linux, running as PDC.

Planning to replace this with 4.5 as AD-DC.

Post by Rowland Penny via samba
I do not use ZFS, mainly because you cannot create a Samba AD DC on it.

Does it fail only when ZFS is used as root filesystem, or for SYSVOL, or
does it not even support ZFS just for shared files?
Not even with "zfs set acltype=posixacl pool/share"?
The root filesystem is ext4.

Klaus

[Rowland Penny via samba]

On Mon, 6 Mar 2017 17:17:07 +0100

Post by Klaus Hartnegg via samba
Does it fail only when ZFS is used as root filesystem, or for SYSVOL,
or does it not even support ZFS just for shared files?
Not even with "zfs set acltype=posixacl pool/share"?
The root filesystem is ext4.

I tried everything I could find on the internet (so I could have missed
something), but I couldn't get Samba to provision as an AD DC, YMMV.

I was using ZFS on Freebsd and could only use UFS, but this also has
problems, mainly in trying to get later versions of Samba to compile.

Rowland

[Andrew Walker via samba]

I don't believe that ZFS on FreeBSD supports acltype=posixacl and sa-based
xattrs.

For the record, the latest version of samba that I've been successfully
able to compile on FreeBSD 10.3 is Samba 4.5.5.

On Mon, Mar 6, 2017 at 11:09 AM, Rowland Penny via samba <

Post by Rowland Penny via samba
On Mon, 6 Mar 2017 17:17:07 +0100

Post by Klaus Hartnegg via samba
Does it fail only when ZFS is used as root filesystem, or for SYSVOL,
or does it not even support ZFS just for shared files?
Not even with "zfs set acltype=posixacl pool/share"?
The root filesystem is ext4.

I tried everything I could find on the internet (so I could have missed
something), but I couldn't get Samba to provision as an AD DC, YMMV.
I was using ZFS on Freebsd and could only use UFS, but this also has
problems, mainly in trying to get later versions of Samba to compile.
Rowland
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba

[Niels Dettenbach (Syndicat IT & Internet) via samba]

Post by Andrew Walker via samba
I don't believe that ZFS on FreeBSD supports acltype=posixacl and sa-based

If you want to see a working samba / zfs / FreeBSD AD / PDC "ecosystem" i recommend just trying and exploring the open FreeBSD based "FreeNAS" which offers / allows exactly this.

We run several Samba 4 with ACL / AD (managed by MS RSAT) on FreeNAS, which anything held on ZFS (except a read only boot partition on USB / flash usually. FreeNAS uses some kind of ntfsv4 acls on ZFS which have "similiar" features, but a bit other semantics, to "emulate" acls for samba - works well.

The current FreeNAS does not providing setting up a AD/PDC by the web GUI (but is planned) afaik, but there are some howtos which decsribe the steps by hand.

Possibly this helps in understanding the concept regarding usage in your own FreeBSD installation.

hth a bit,

Niels.

[Rowland Penny via samba]

On Sat, 11 Mar 2017 08:18:26 +0100
"Niels Dettenbach \(Syndicat IT & Internet\) via samba"

Am 11. März 2017 00:04:13 MEZ schrieb Andrew Walker via samba

Post by Andrew Walker via samba
I don't believe that ZFS on FreeBSD supports acltype=posixacl and sa-based

If you want to see a working samba / zfs / FreeBSD AD / PDC
"ecosystem" i recommend just trying and exploring the open FreeBSD
based "FreeNAS" which offers / allows exactly this.
We run several Samba 4 with ACL / AD (managed by MS RSAT) on FreeNAS,
which anything held on ZFS (except a read only boot partition on
USB / flash usually. FreeNAS uses some kind of ntfsv4 acls on ZFS
which have "similiar" features, but a bit other semantics, to
"emulate" acls for samba - works well.

From what I can see, Freenas is using Samba 4.3.6 and if it is using
ZFS, it is probably also using the ntvfs filesystem instead of 3fs.
This could be a problem with Freenas 10, it uses a later version of
Samba, where by default, ntvfs is turned off.

Samba does not understand ntvfsv4 ACLs, this is where the problem lies,
you cannot provision Samba as an AD DC on ZFS.

Also, there is no such thing as an AD/PDC, a PDC is an NT4-style domain
controller. All AD domain controllers are equal, so there is no such
concept as an AD primary DC. I suppose you could refer to the initial
DC as 'The First Domain Controller I Set UP', but do you really want to
call it a 'TFDCISU' ?? ;-)

Rowland

[Andrew Walker via samba]

FreeNAS 9.10.2-STABLE has Samba 4.5.5. By default "zfsacl" is enabled on
all samba shares. Additionally the zfs "aclmode" property is set to
"restricted" on all samba shares (except I believe the sysvol share). I
believe on the sysvol dataset / share, the aclmode is set to "passthrough"
and "zfsacl" is not enabled. I do not use freenas as and ADDC and so can't
comment about specific configuration changes that are made in that regard,
but I am happy to speculate that they probably handle ACLs on the sysvol
share via acl_xattr while the underlying filesystem happily pretends it's a
normal Unix FS. Jordan Hubbard recently wrote a decent summary of the
choices FreeNAS made with respect to handling ACLs here:
https://forums.freenas.org/index.php?threads/update-on-smb-permissions-docker-containers-zfs-a-brief-history-of-posix-perms-etc.51272/

The source for the FreeNAS 9.10l.2 script that generates its smb.conf file
is here:
https://github.com/freenas/freenas/blob/9.10.2-STABLE/src/freenas/usr/local/libexec/nas/generate_smb4_conf.py

I do vaguely recall some problems people had when they tried to place the
sysvol share on a dataset with the aclmode set to "restricted"... well, not
the exact problems, but rather a general wailing and gnashing of teeth.

On Sat, Mar 11, 2017 at 2:50 AM, Rowland Penny via samba <

Post by Rowland Penny via samba
On Sat, 11 Mar 2017 08:18:26 +0100
"Niels Dettenbach \(Syndicat IT & Internet\) via samba"

Post by Niels Dettenbach (Syndicat IT & Internet) via samba

Post by Andrew Walker via samba
I don't believe that ZFS on FreeBSD supports acltype=posixacl and sa-based

If you want to see a working samba / zfs / FreeBSD AD / PDC
"ecosystem" i recommend just trying and exploring the open FreeBSD
based "FreeNAS" which offers / allows exactly this.
We run several Samba 4 with ACL / AD (managed by MS RSAT) on FreeNAS,
which anything held on ZFS (except a read only boot partition on
USB / flash usually. FreeNAS uses some kind of ntfsv4 acls on ZFS
which have "similiar" features, but a bit other semantics, to
"emulate" acls for samba - works well.

From what I can see, Freenas is using Samba 4.3.6 and if it is using
ZFS, it is probably also using the ntvfs filesystem instead of 3fs.
This could be a problem with Freenas 10, it uses a later version of
Samba, where by default, ntvfs is turned off.
Samba does not understand ntvfsv4 ACLs, this is where the problem lies,
you cannot provision Samba as an AD DC on ZFS.
Also, there is no such thing as an AD/PDC, a PDC is an NT4-style domain
controller. All AD domain controllers are equal, so there is no such
concept as an AD primary DC. I suppose you could refer to the initial
DC as 'The First Domain Controller I Set UP', but do you really want to
call it a 'TFDCISU' ?? ;-)
Rowland
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba

[David Disseldorp via samba]

Hi Brian,

Post by brian lamb via samba
Configured as you mention, and prior with the other tweaks I had in the first place, Im getting the same results, which is "FSUTIL utility requires a Local NTFS Volume", and mapping it.

Hmm, this works fine for me against a Samba 4.4.2 Btrfs backed Samba
share.

smb.conf:
[global]
...
store dos attributes = yes

[share]
path = /tmp/tmp.UR3ZbULpHJ
read only = no

Windows Server 2012 client:
("store dos attributes = no" configured on server)
PS> fsutil.exe sparse setflag \\samba_server\share\file
Error: Incorrect function.

("store dos attributes = yes" configured on server)
PS> fsutil.exe sparse setflag \\samba_server\share\file
PS> fsutil.exe sparse queryflag \\samba_server\share\file
This file is set as sparse

I don't think it's worth trying to triage your application problems
any further until you have this simple test case working. Please set
"log level = 10" on the Samba server, and rerun the above test.
Attach the resulting debug log to a bugzilla.samba.org ticket.

Also, please check whether you can manually set an extended attribute
locally against a file in your share path. e.g.
samba_server:/tmp/tmp.UR3ZbULpHJ/:[0]# setfattr -n user.stuff -v xattr-data file
samba_server:/tmp/tmp.UR3ZbULpHJ/:[0]# getfattr -d file
user.stuff="xattr-data"

Furthermore, could you please check whether Samba is compiled with xattr
support - i.e.:
samba_server:# smbd -b | grep -i xattr

Post by brian lamb via samba
I then went to contig, `contig -n X:\share\sparesefile 0`, as well as a few other size. Maybe im using it wrong, but some testing on contig has only "Access is denied" when I try to write to anything.
Oh one change, I put these settings only in Global (only, as opposed to also in the pool definition), I just realized. I believe global should be enough shouldn’t it?

[globa]l section is fine, as long as it's not also set in the share
section.

Post by brian lamb via samba
[global]
# ZFS SPARSE
#share:fake_fscaps = 0
store dos attributes = yes
strict allocate = yes
I hadn’t heard any mention of the importance of strict allocate, would that setting be wise then?

It doesn't affect how FSCTL_SET_SPARE is handled.

Post by brian lamb via samba
What are some other tools I might try that with? I can try straight on the linux too, just as a test, if there is any tools recommended to test that with. I will also look something up for that tomorrow as well.

Let's get "fsutil.exe sparse setflag" working first.

Cheers, David

[brian lamb via samba]

For some reason It's not erroring now, I will look into it more, but I suspect it’s the settings, the last thing mentioned
David

[globa]l section is fine, as long as it's not also set in the share section.

So, yeah, that may have fixed im for the time being not getting the sparse errors, for the first time in 4 months, however, there is a couple other problems, so hopefully they wont come back after these disappear. To be specific the latest issues can be seen by the snippet below on the Freefilesync log. Im trying to dial in the symlink operation, and any other remaining errors.

This demonstrates two issues
1) Cannot write file Error Code 2: The system cannot find the file specified. (BackupWrite)
2) Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)

The symbolic link setting I think I tried both ways, I think I have another toggle on the 3rd party app(ffs) which I need to also have in the correct setting, and to get the combination correct. Im not sure what the problem is on the BackupWrite. I had a perminssion error at first, I `chown`ed the dir, and that went away, I still see the word BackupWrite, which I haven’t seen before, not sure its origin.


1:00:58 AM Error Cannot write file "\\FALCON\pool\appletree\data\public\Downloads\photoshop lightroom plugins\Totally RAD LR Presets\PRE.zip.ffs_tmp".
Error Code 2: The system cannot find the file specified. (BackupWrite)
1:04:22 AM Error Cannot write file "\\FALCON\pool\appletree\data\public\Games\Flight Sims\F4\Falcon_BMS_4.33_U1_Setup.zip.ffs_tmp".
Error Code 2: The system cannot find the file specified. (BackupWrite)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\Application Data" to
"\\FALCON\pool\appletree\data\users\Guest\Application Data".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\Cookies" to
"\\FALCON\pool\appletree\data\users\Guest\Cookies".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\Local Settings" to
"\\FALCON\pool\appletree\data\users\Guest\Local Settings".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\My Documents" to
"\\FALCON\pool\appletree\data\users\Guest\My Documents".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\NetHood" to
"\\FALCON\pool\appletree\data\users\Guest\NetHood".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\PrintHood" to
"\\FALCON\pool\appletree\data\users\Guest\PrintHood".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\Recent" to
"\\FALCON\pool\appletree\data\users\Guest\Recent".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\SendTo" to
"\\FALCON\pool\appletree\data\users\Guest\SendTo".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\Start Menu" to
"\\FALCON\pool\appletree\data\users\Guest\Start Menu".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\Templates" to
"\\FALCON\pool\appletree\data\users\Guest\Templates".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\AppData\Local\Application Data" to
"\\FALCON\pool\appletree\data\users\Guest\AppData\Local\Application Data".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\AppData\Local\History" to
"\\FALCON\pool\appletree\data\users\Guest\AppData\Local\History".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW))


1:00:58 AM Error Cannot write file "\\FALCON\pool\appletree\data\public\Downloads\photoshop lightroom plugins\SLR Lounge - Lightroom Preset System v5.1 - PC [lessgravity]\LR Preset System v5.1 Windows Installer.exe.ffs_tmp".
Error Code 2: The system cannot find the file specified. (BackupWrite)
1:00:58 AM Error Cannot write file "\\FALCON\pool\appletree\data\public\Downloads\photoshop lightroom plugins\Totally RAD LR Presets\PRE.zip.ffs_tmp".
Error Code 2: The system cannot find the file specified. (BackupWrite)
1:04:22 AM Error Cannot write file "\\FALCON\pool\appletree\data\public\Games\Flight Sims\F4\Falcon_BMS_4.33_U1_Setup.zip.ffs_tmp".
Error Code 2: The system cannot find the file specified. (BackupWrite)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\Application Data" to
"\\FALCON\pool\appletree\data\users\Guest\Application Data".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\Cookies" to
"\\FALCON\pool\appletree\data\users\Guest\Cookies".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\Local Settings" to
"\\FALCON\pool\appletree\data\users\Guest\Local Settings".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\My Documents" to
"\\FALCON\pool\appletree\data\users\Guest\My Documents".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\NetHood" to
"\\FALCON\pool\appletree\data\users\Guest\NetHood".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\PrintHood" to
"\\FALCON\pool\appletree\data\users\Guest\PrintHood".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\Recent" to
"\\FALCON\pool\appletree\data\users\Guest\Recent".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\SendTo" to
"\\FALCON\pool\appletree\data\users\Guest\SendTo".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\Start Menu" to
"\\FALCON\pool\appletree\data\users\Guest\Start Menu".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\Templates" to
"\\FALCON\pool\appletree\data\users\Guest\Templates".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\AppData\Local\Application Data" to
"\\FALCON\pool\appletree\data\users\Guest\AppData\Local\Application Data".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)
1:04:22 AM Error Cannot copy symbolic link
"D:\users\Guest\AppData\Local\History" to
"\\FALCON\pool\appletree\data\users\Guest\AppData\Local\History".
Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)

So, if anybody sees anything related, let me know,
and THANKS MUCH otherwise, for all the support, I hadn’t seen the dos attributes option until I had already posted my first message, but still, its not always apparent if I should actually put this setting in global, I think I might try it just in the definition for the zfs share next though, that would be ideal. The problem is, I think it took a full reboot, or maybe even zfs export and import, or something to actually trigger in these new settings, something changed at an unexpected time, that’s all I know, the reason I say this is because things just didn’t start working after a samba config update and a `systemctl restart smb && systemctl restart nmb`, it was hours or days later on another test which I noticed it started working.


Regards,
Brian






-----Original Message-----
From: samba [mailto:samba-***@lists.samba.org] On Behalf Of David Disseldorp via samba
Sent: Thursday, February 16, 2017 4:54 AM
To: brian lamb via samba <***@lists.samba.org>
Subject: Re: [Samba] (DeviceIoControl, FSCTL_SET_SPARSE)

Hi Brian,

Configured as you mention, and prior with the other tweaks I had in the first place, Im getting the same results, which is "FSUTIL utility requires a Local NTFS Volume", and mapping it.

Hmm, this works fine for me against a Samba 4.4.2 Btrfs backed Samba share.

smb.conf:
[global]
...
store dos attributes = yes

[share]
path = /tmp/tmp.UR3ZbULpHJ
read only = no

Windows Server 2012 client:
("store dos attributes = no" configured on server)
PS> fsutil.exe sparse setflag \\samba_server\share\file
Error: Incorrect function.

("store dos attributes = yes" configured on server)
PS> fsutil.exe sparse setflag \\samba_server\share\file fsutil.exe
PS> sparse queryflag \\samba_server\share\file
This file is set as sparse

I don't think it's worth trying to triage your application problems any further until you have this simple test case working. Please set "log level = 10" on the Samba server, and rerun the above test.
Attach the resulting debug log to a bugzilla.samba.org ticket.

Also, please check whether you can manually set an extended attribute locally against a file in your share path. e.g.
samba_server:/tmp/tmp.UR3ZbULpHJ/:[0]# setfattr -n user.stuff -v xattr-data file samba_server:/tmp/tmp.UR3ZbULpHJ/:[0]# getfattr -d file user.stuff="xattr-data"

Furthermore, could you please check whether Samba is compiled with xattr support - i.e.:
samba_server:# smbd -b | grep -i xattr

I then went to contig, `contig -n X:\share\sparesefile 0`, as well as a few other size. Maybe im using it wrong, but some testing on contig has only "Access is denied" when I try to write to anything.
Oh one change, I put these settings only in Global (only, as opposed to also in the pool definition), I just realized. I believe global should be enough shouldn’t it?

[globa]l section is fine, as long as it's not also set in the share section.

[global]
# ZFS SPARSE
#share:fake_fscaps = 0
store dos attributes = yes
strict allocate = yes
I hadn’t heard any mention of the importance of strict allocate, would that setting be wise then?

It doesn't affect how FSCTL_SET_SPARE is handled.

What are some other tools I might try that with? I can try straight on the linux too, just as a test, if there is any tools recommended to test that with. I will also look something up for that tomorrow as well.

Let's get "fsutil.exe sparse setflag" working first.

Cheers, David

--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba

[David Disseldorp via samba]

Hi Brian,

Post by brian lamb via samba
For some reason It's not erroring now, I will look into it more, but I suspect it’s the settings, the last thing mentioned

Glad to hear.

Post by brian lamb via samba
David

[globa]l section is fine, as long as it's not also set in the share section.

So, yeah, that may have fixed im for the time being not getting the sparse errors, for the first time in 4 months, however, there is a couple other problems, so hopefully they wont come back after these disappear. To be specific the latest issues can be seen by the snippet below on the Freefilesync log. Im trying to dial in the symlink operation, and any other remaining errors.
This demonstrates two issues
1) Cannot write file Error Code 2: The system cannot find the file specified. (BackupWrite)
2) Error Code 183: Cannot create a file when that file already exists. (CreateSymbolicLinkW)

Hmm, it looks as though your application uses some exotic functions
which aren't supported by Samba at a protocol level:
- https://msdn.microsoft.com/en-us/library/aa363866(VS.85).aspx
- https://msdn.microsoft.com/en-us/library/aa362511(v=vs.85).aspx

Please capture network traffic, once with the application running
against a Samba server, and once against a (recent) Windows SMB2+
server:
https://wiki.samba.org/index.php/Capture_Packets

When you have the captures, please create a bug at bugzilla.samba.org,
and attach them there.

You should obtain these captures in an isolated test network, with test
data - they may be seen by others.

Cheers, David

[brian lamb via samba]

Hi Again,

Doing some testing just now with `dd if=/dev/zero of=./sparser.img bs=1 count=0 seek=0` and `truncate -s 0 asparsefile` with no issues on the share, but not across samba. I was thinking, what If I mounted that to /mnt using cifs, and try to use the linux to use the same two methods above as a test. Ill probably will have done that by the next reply, however any thoughts on that are welcome.

Brian

-----Original Message-----
From: ***@wolfdogg.org [mailto:***@wolfdogg.org]
Sent: Thursday, February 16, 2017 1:21 AM
To: '***@lists.samba.org' <***@lists.samba.org>
Subject: RE: [Samba] (DeviceIoControl, FSCTL_SET_SPARSE)

Hi David,

Configured as you mention, and prior with the other tweaks I had in the first place, Im getting the same results, which is "FSUTIL utility requires a Local NTFS Volume", and mapping it.

I then went to contig, `contig -n X:\share\sparesefile 0`, as well as a few other size. Maybe im using it wrong, but some testing on contig has only "Access is denied" when I try to write to anything.

Oh one change, I put these settings only in Global (only, as opposed to also in the pool definition), I just realized. I believe global should be enough shouldn’t it?

[global]
# ZFS SPARSE
#share:fake_fscaps = 0
store dos attributes = yes
strict allocate = yes

I hadn’t heard any mention of the importance of strict allocate, would that setting be wise then?

What are some other tools I might try that with? I can try straight on the linux too, just as a test, if there is any tools recommended to test that with. I will also look something up for that tomorrow as well.

Thanks for the info, much appreciated.
-Brian

-----Original Message-----
From: samba [mailto:samba-***@lists.samba.org] On Behalf Of David Disseldorp via samba
Sent: Wednesday, February 15, 2017 5:23 AM
To: brian lamb via samba <***@lists.samba.org>
Subject: Re: [Samba] (DeviceIoControl, FSCTL_SET_SPARSE)

Post by brian lamb via samba
Thanks for your response.
Cannot write file attributes of "\\MYHOST\backup\sparse\My Family Tree 6.0.zip.ffs_tmp".
Error Code 1: Incorrect function. (DeviceIoControl, FSCTL_SET_SPARSE)

Hmm, that seems to indicate that the initial fsctl request is failing.
Could you please try manually setting a file on the Samba share as sparse with "fsutil sparse setflag <file_on_samba_share>":
https://technet.microsoft.com/en-us/library/cc788025(v=ws.11).aspx

Does it succeed when Samba is configured as per below?

Post by brian lamb via samba
Samba version 4.4.4
PID Username Group Machine Protocol Version Encryption Signing
----------------------------------------------------------------------------------------------------------------------------------------
21582 backupadmin sys1 myhost (ipv4:192.168.1.99:55068) SMB2_10 - -
I just tried the option " store dos attributes = no"
Im getting the same error as above still unfortunately, but im doing
some more thorough testing on that. WOud you happen to know if I want
the other two options set, and to what, at all? (> strict allocate &
share:fake_fscaps = 0)

Given that the application is complaining about FSCTL_SET_SPARSE failure, I would guess that it requires sparse-file support. Please set "store dos attributes = yes" and drop the fake_fscaps setting.

Cheers, David

--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba