[Samba] Performance Problem / failed to verify PAC server signature

[Christoph Kaegi]

Hello List

We run a Solaris9 Server running Samba 3.0.20, Local Users (no winbind)
but authenticating against ADS.
There are up to 800 concurrent users, mostly Windows XP SP3.

When clients access MyDocuments, which is redirected to the Samba
share, we observe several

"Session Setup AndX Request"s

followed by

"Session Setup AndX Response, Error: STATUS_LOGON_FAILURE"s

The delay between the request and the negative response is negligible
when less than 200 users are online. But at more than 500 concurrent
users, the delay becomes something between 1 to 5 secons.

This delays access to MyDocuments quite a bit, considering that
there are sometimes up to 10 such requests.

So I'm interested in finding the problem and fixing it.
The log says:

-------------------------------------- 8< --------------------------------------
[2005/11/21 16:09:28, 3] libsmb/clikrb5.c:smb_krb5_verify_checksum(695)
smb_krb5_verify_checksum: krb5_c_verify_checksum() failed: Bad encryption type
[2005/11/21 16:09:28, 2] libads/authdata.c:check_pac_checksum(666)
check_pac_checksum: PAC Verification failed: Bad encryption type (-1765328196)
[2005/11/21 16:09:28, 0] libads/authdata.c:decode_pac_data(876)
decode_pac_data: failed to verify PAC server signature
[2005/11/21 16:09:28, 3] libads/kerberos_verify.c:ads_verify_ticket(416)
ads_verify_ticket: failed to decode PAC_DATA: NT_STATUS_ACCESS_DENIED
-------------------------------------- 8< --------------------------------------

Other relevant Server settings are:

-------------------------------------- 8< --------------------------------------
security = ADS
realm = FOO.BAR
use kerberos keytab = yes
workgroup = FOOBAR

log file = /var/log/samba/smbd.log
log level = 10
max log size = 0
socket options = TCP_NODELAY
local master = no
domain master = no
preferred master = no
domain logons = no
wins support = no
-------------------------------------- 8< --------------------------------------

Any hints?

Thanks alot

Christoph

[Guenther Deschner]

Hi,

Post by Christoph Kaegi
Hello List
We run a Solaris9 Server running Samba 3.0.20, Local Users (no winbind)
but authenticating against ADS.
There are up to 800 concurrent users, mostly Windows XP SP3.
When clients access MyDocuments, which is redirected to the Samba
share, we observe several
"Session Setup AndX Request"s
followed by
"Session Setup AndX Response, Error: STATUS_LOGON_FAILURE"s
The delay between the request and the negative response is negligible
when less than 200 users are online. But at more than 500 concurrent
users, the delay becomes something between 1 to 5 secons.
This delays access to MyDocuments quite a bit, considering that
there are sometimes up to 10 such requests.
So I'm interested in finding the problem and fixing it.
-------------------------------------- 8< --------------------------------------
[2005/11/21 16:09:28, 3] libsmb/clikrb5.c:smb_krb5_verify_checksum(695)
smb_krb5_verify_checksum: krb5_c_verify_checksum() failed: Bad encryption type
[2005/11/21 16:09:28, 2] libads/authdata.c:check_pac_checksum(666)
check_pac_checksum: PAC Verification failed: Bad encryption type (-1765328196)
[2005/11/21 16:09:28, 0] libads/authdata.c:decode_pac_data(876)
decode_pac_data: failed to verify PAC server signature
[2005/11/21 16:09:28, 3] libads/kerberos_verify.c:ads_verify_ticket(416)
ads_verify_ticket: failed to decode PAC_DATA: NT_STATUS_ACCESS_DENIED
-------------------------------------- 8< --------------------------------------

First of all: are you sure you are running Samba 3.0.20? The PAC
verification code is not in any of the 3.0.20/a/b tarball releases (just
accidentially in the 3.0.20a subversion tags directory) but only in the
3.0.21 series of pre-releases/rcs.

Then you most probably are forced to use DES keys when authenticating with
Kerberos on your OS, right? PAC verification must then fail due to a bug
in Windows (which fails to put DES-based checksum into the PAC
signatures), so we can't verify the signature. What exact Kerberos library
are you using (version) ?

Nonetheless, failure of the PAC verification is non-critical, we just
return to old behaviour and ignore the PAC again, meaning that you can
ignore the error messages.

Guenther

[Christoph Kaegi]

Post by Guenther Deschner

Post by Christoph Kaegi
-------------------------------------- 8< --------------------------------------
[2005/11/21 16:09:28, 3] libsmb/clikrb5.c:smb_krb5_verify_checksum(695)
smb_krb5_verify_checksum: krb5_c_verify_checksum() failed: Bad encryption type
[2005/11/21 16:09:28, 2] libads/authdata.c:check_pac_checksum(666)
check_pac_checksum: PAC Verification failed: Bad encryption type (-1765328196)
[2005/11/21 16:09:28, 0] libads/authdata.c:decode_pac_data(876)
decode_pac_data: failed to verify PAC server signature
[2005/11/21 16:09:28, 3] libads/kerberos_verify.c:ads_verify_ticket(416)
ads_verify_ticket: failed to decode PAC_DATA: NT_STATUS_ACCESS_DENIED
-------------------------------------- 8< --------------------------------------

First of all: are you sure you are running Samba 3.0.20? The PAC
verification code is not in any of the 3.0.20/a/b tarball releases (just
accidentially in the 3.0.20a subversion tags directory) but only in the
3.0.21 series of pre-releases/rcs.

The production Server runs 3.0.20, but the test Server, where I
analyzed this and where the logs are coming from is 3.0.21rc1 indeed.

Sorry for the confusion.

But in both cases, the behaviour on the network is the same
(STATUS_LOGON_FAILUREs with a certain delay, depending on load)

Post by Guenther Deschner
Then you most probably are forced to use DES keys when authenticating with
Kerberos on your OS, right? PAC verification must then fail due to a bug
in Windows (which fails to put DES-based checksum into the PAC
signatures), so we can't verify the signature. What exact Kerberos library
are you using (version) ?

Hm, how can I determine, if I use DES keys? I have the following in
krb5.conf (if that is what you mean):

-------------------------------------- 8< --------------------------------------
default_tkt_enctypes = des-cbc-crc, des-cbc-md5
default_tgs_enctypes = des-cbc-crc, des-cbc-md5
-------------------------------------- 8< --------------------------------------

I derived this from google knowledge, but I'll change this
gladly if you tell me it is wrong.

Kerberos is MIT Kerbers5 1.4

Post by Guenther Deschner
Nonetheless, failure of the PAC verification is non-critical, we just
return to old behaviour and ignore the PAC again, meaning that you can
ignore the error messages.

Yes, everything is functioning, but we have delays of several minutes
for some clients, which is not really acceptable for them anymore.

These sesssion setup requests / failures are responsible for the most
part of the time it takes to access MyDocuments. So I figured, if
we can solve this, the delays should be back in acceptable ranges.

What exactly is this PAC, btw.?

Thanks very much

Chris

[Doug VanLeuven]

Post by Christoph Kaegi

Post by Guenther Deschner

Post by Christoph Kaegi
-------------------------------------- 8< --------------------------------------
[2005/11/21 16:09:28, 3] libsmb/clikrb5.c:smb_krb5_verify_checksum(695)
smb_krb5_verify_checksum: krb5_c_verify_checksum() failed: Bad encryption type
[2005/11/21 16:09:28, 2] libads/authdata.c:check_pac_checksum(666)
check_pac_checksum: PAC Verification failed: Bad encryption type (-1765328196)
[2005/11/21 16:09:28, 0] libads/authdata.c:decode_pac_data(876)
decode_pac_data: failed to verify PAC server signature
[2005/11/21 16:09:28, 3] libads/kerberos_verify.c:ads_verify_ticket(416)
ads_verify_ticket: failed to decode PAC_DATA: NT_STATUS_ACCESS_DENIED
-------------------------------------- 8< --------------------------------------

First of all: are you sure you are running Samba 3.0.20? The PAC
verification code is not in any of the 3.0.20/a/b tarball releases (just
accidentially in the 3.0.20a subversion tags directory) but only in the
3.0.21 series of pre-releases/rcs.

The production Server runs 3.0.20, but the test Server, where I
analyzed this and where the logs are coming from is 3.0.21rc1 indeed.
Sorry for the confusion.
But in both cases, the behaviour on the network is the same
(STATUS_LOGON_FAILUREs with a certain delay, depending on load)

Post by Guenther Deschner
Then you most probably are forced to use DES keys when authenticating with
Kerberos on your OS, right? PAC verification must then fail due to a bug
in Windows (which fails to put DES-based checksum into the PAC
signatures), so we can't verify the signature. What exact Kerberos library
are you using (version) ?

Hm, how can I determine, if I use DES keys? I have the following in
-------------------------------------- 8< --------------------------------------
default_tkt_enctypes = des-cbc-crc, des-cbc-md5
default_tgs_enctypes = des-cbc-crc, des-cbc-md5
-------------------------------------- 8< --------------------------------------
I derived this from google knowledge, but I'll change this
gladly if you tell me it is wrong.
Kerberos is MIT Kerbers5 1.4

With Kerberos 1.4 you should include rc4-hmac in the list of enctypes.
It is the native mode of windows.

Regards, Doug

[Christoph Kaegi]

Post by Doug VanLeuven

Post by Christoph Kaegi
Hm, how can I determine, if I use DES keys? I have the following in
-------------------------------------- 8<
--------------------------------------
default_tkt_enctypes = des-cbc-crc, des-cbc-md5
default_tgs_enctypes = des-cbc-crc, des-cbc-md5
-------------------------------------- 8<
--------------------------------------
I derived this from google knowledge, but I'll change this
gladly if you tell me it is wrong.
Kerberos is MIT Kerbers5 1.4

With Kerberos 1.4 you should include rc4-hmac in the list of enctypes.
It is the native mode of windows.

Thanks! I added this to my /etc/krb5.conf.

It didn't help my problem though.

Any more hints?

Thanks
Chris

[Doug VanLeuven]

Post by Christoph Kaegi

Post by Doug VanLeuven

Post by Christoph Kaegi
Hm, how can I determine, if I use DES keys? I have the following in
-------------------------------------- 8<
--------------------------------------
default_tkt_enctypes = des-cbc-crc, des-cbc-md5
default_tgs_enctypes = des-cbc-crc, des-cbc-md5
-------------------------------------- 8<
--------------------------------------
I derived this from google knowledge, but I'll change this
gladly if you tell me it is wrong.
Kerberos is MIT Kerbers5 1.4

With Kerberos 1.4 you should include rc4-hmac in the list of enctypes.
It is the native mode of windows.

Thanks! I added this to my /etc/krb5.conf.
It didn't help my problem though.
Any more hints?

Well, no. Maybe. Yes. Been a while since I confronted moving
between des & arc4.

in source/libads/ldap.c
#ifndef ENCTYPE_ARCFOUR_HMAC
acct_control |= UF_USE_DES_KEY_ONLY;
#endif

So my experience is if it is defined in the include file at compile
time, all accounts are created arc4 capable. I don't see any
flags in the "smbd -b" build options that confirm this either way

You could check (if your computer joined a long time ago) using ldp.exe
which translates the flags into english.
cn=<your computer>,cn=Computers,cn=<your domain>
userAccountControl
0x00200000 ADS_UF_USE_DES_KEY_ONLY
Subtract that out to clear the bit, if set, which it might be, depending
on how long ago you joined. I use adsiedit.msc which presents the
flag in decimal.

Also, I use this samba option:
use kerberos keytab = yes
Which means samba creates /etc/krb5.keytab entries for you when you
join the domain.
If you use that option, your keytab file will probably only have des
entries in it from when you joined and only des-cbc-crc and des-cbc-md5
were allowed.

You can generate a new set of keys with this command
net ads changetrustpw
Or you can delete the computer account on the domain controller,
delete the existing keytab entries (or the keytab file if no other
keys are present), and rejoin the domain.
I got leary of changetrustpw because sometimes it would hang in
the kerberos libraries in version krb5 1.3.4 if one ran the command
several times in a short period of time. Once was always OK.
FWIW I believe it hangs trying to delete older key versions that
are still current and in use.

If you don't use that option and you manually created the keytab entry
with ktpass.exe, then you'd know if you used
/DesOnly
and if you did, you'd need to cut a new keytab that wasn't des only.

Any way you edit or update, there is caching of tickets going on.
I remember after one long night of updating kerberos and rejoining the
domain, nothing worked. The machine beat me. Next day I thought
lets try it one more time. Worked like a charm without modification.
Kerberos hammered home the lesson that patience is a virtue.
Never did come up with a deterministic method to compute the time
to outlive the cache. Don't know if it's cached in samba secrets
or the windows domain controller.

Luck, Doug

[Christoph Kaegi]

Post by Doug VanLeuven
Well, no. Maybe. Yes. Been a while since I confronted moving
between des & arc4.
in source/libads/ldap.c
#ifndef ENCTYPE_ARCFOUR_HMAC
acct_control |= UF_USE_DES_KEY_ONLY;
#endif

I have in source/include/config.h:

/* Whether the ENCTYPE_ARCFOUR_HMAC_MD5 key type is available */
/* #undef HAVE_ENCTYPE_ARCFOUR_HMAC_MD5 */

And my MIT 1.4 says in krb5.h:

[...]
#define CKSUMTYPE_HMAC_SHA1_96_AES128 0x000f
#define CKSUMTYPE_HMAC_SHA1_96_AES256 0x0010
#define CKSUMTYPE_HMAC_MD5_ARCFOUR -138 /*Microsoft md5 hmac cksumtype*/
[...]

That last define of CKSUMTYPE_HMAC_MD5_ARCFOUR seems doesn't
look promising.

Does that mean, that my Kerberos library doesn't support
the encryption type that I need? (I checked also krb5-1.4.3,
which has the same definition)

Post by Doug VanLeuven
So my experience is if it is defined in the include file at compile
time, all accounts are created arc4 capable. I don't see any
flags in the "smbd -b" build options that confirm this either way

What is an arc4 capable Unix account?

Post by Doug VanLeuven
use kerberos keytab = yes
Which means samba creates /etc/krb5.keytab entries for you when you
join the domain.
If you use that option, your keytab file will probably only have des
entries in it from when you joined and only des-cbc-crc and des-cbc-md5
were allowed.

I rejoined, deleted the AD computer account, recreated it several
times.

All funny things are happening, including:

-------------------------------------- 8< --------------------------------------
[2005/11/23 14:32:47, 0] lib/fault.c:fault_report(36)
===============================================================
[2005/11/23 14:32:47, 0] lib/fault.c:fault_report(37)
INTERNAL ERROR: Signal 11 in pid 20569 (3.0.21rc1)
Please read the Trouble-Shooting section of the Samba3-HOWTO
[2005/11/23 14:32:47, 0] lib/fault.c:fault_report(39)

From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
[2005/11/23 14:32:47, 0] lib/fault.c:fault_report(40)
===============================================================
[2005/11/23 14:32:47, 0] lib/util.c:smb_panic2(1554)
PANIC: internal error
-------------------------------------- 8< --------------------------------------

after a successful join...

Chris

[Doug VanLeuven]

Post by Christoph Kaegi

Post by Doug VanLeuven
Well, no. Maybe. Yes. Been a while since I confronted moving
between des & arc4.
in source/libads/ldap.c
#ifndef ENCTYPE_ARCFOUR_HMAC
acct_control |= UF_USE_DES_KEY_ONLY;
#endif

/* Whether the ENCTYPE_ARCFOUR_HMAC_MD5 key type is available */
/* #undef HAVE_ENCTYPE_ARCFOUR_HMAC_MD5 */

Heimdal kerberos defines rc4-hmac this way.

From MIT site:
Supported Encryption Types
arcfour-hmac
rc4-hmac
arcfour-hmac-md5
RC4 with HMAC/MD5
These are all synonyms.

Post by Christoph Kaegi
[...]
#define CKSUMTYPE_HMAC_SHA1_96_AES128 0x000f
#define CKSUMTYPE_HMAC_SHA1_96_AES256 0x0010
#define CKSUMTYPE_HMAC_MD5_ARCFOUR -138 /*Microsoft md5 hmac cksumtype*/
[...]
That last define of CKSUMTYPE_HMAC_MD5_ARCFOUR seems doesn't
look promising.

About 20 lines before that you should see
#define ENCTYPE_ARCFOUR_HMAC 0x0017
#define ENCTYPE_ARCFOUR_HMAC_EXP 0x0018

So your compiled samba will have rc4-hmac support.

Post by Christoph Kaegi
Does that mean, that my Kerberos library doesn't support
the encryption type that I need? (I checked also krb5-1.4.3,
which has the same definition)

Post by Doug VanLeuven
So my experience is if it is defined in the include file at compile
time, all accounts are created arc4 capable. I don't see any
flags in the "smbd -b" build options that confirm this either way

What is an arc4 capable Unix account?

Not arc4 capable Unix, arc4-hmac capable Windows computer account.
This is the default state of windows accounts and a flag is neccessary
to force des-only usage.

In Active Directory
In domain
wherever computer accounts get setup
cn=<computer name>
userAccountControl: 0x11000
The two set bits mean:
UF_WORKSTATION_TRUST_ACCOUNT | UF_DONT_EXPIRE_PASSWD
This account is arc4 capable.
An account that is des only has this value:
userAccountControl: 0x211000
UF_WORKSTATION_TRUST_ACCOUNT | UF_DONT_EXPIRE_PASSWD
| UF_USE_DES_KEY_ONLY

userAccountControl exists in user accounts too.

ktpass.exe:
+des (des only - default for command)
-des (not des only)

Post by Christoph Kaegi

Post by Doug VanLeuven
use kerberos keytab = yes
Which means samba creates /etc/krb5.keytab entries for you when you
join the domain.
If you use that option, your keytab file will probably only have des
entries in it from when you joined and only des-cbc-crc and des-cbc-md5
were allowed.

I rejoined, deleted the AD computer account, recreated it several
times.
-------------------------------------- 8< --------------------------------------
[2005/11/23 14:32:47, 0] lib/fault.c:fault_report(36)
===============================================================
[2005/11/23 14:32:47, 0] lib/fault.c:fault_report(37)
INTERNAL ERROR: Signal 11 in pid 20569 (3.0.21rc1)
Please read the Trouble-Shooting section of the Samba3-HOWTO
[2005/11/23 14:32:47, 0] lib/fault.c:fault_report(39)
From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
[2005/11/23 14:32:47, 0] lib/fault.c:fault_report(40)
===============================================================
[2005/11/23 14:32:47, 0] lib/util.c:smb_panic2(1554)
PANIC: internal error
-------------------------------------- 8< --------------------------------------
after a successful join...

Bummer, shouldn't happen. But it could be the kerberos.

I was curious, so I dragged out an old des only machine
used in testing last year.
RH9 with a custom 1.3.5 MIT kerberos.
Ethereal traces on port 88 show machine using only des
Compiled and installed samba 3.0.21pre3 SVN 11739
Ran it in des-only mode without issue, but had no easy way to check
redirected folders.

Then I converted it to your system of using a ktpass.exe generated
keytab using rc4-hmac.

Stopped samba
edit smb.conf and remove "use kerberos keytab = yes"
Deleted the existing computer account in AD
Deleted the existing mapped user account in AD
Deleted /etc/krb5.keytab
Edit krb5.conf and add rc4-hmac as -first- enctype in list for
default_tgs_enctypes, default_tkt_enctypes, permitted_enctypes
Deleted samba's private.tdb
Deleted samba's winbindd_cache.tdb (just in case)
Created a new windows user account to be used for mapping in ktpass.exe
Ran ktpass.exe on domain controller with "-DesOnly"
Read the new keytab and write /etc/krb5.conf with it
Run "net ads join"
Ethereal trace on port 88 show rc4-hmac negotiated tickets

Using a ktpass.exe generated keytab, the AD computer account and the
AD mapped user account attribute userAccountControl must agree on the
flag UF_USE_DES_KEY_ONLY. They either both indicate it or they
both don't indicate it, but they can't be mixed.

We'll be enjoying Thanksgiving holiday here.
Regards, Doug

[Doug VanLeuven]

Post by Doug VanLeuven
Then I converted it to your system of using a ktpass.exe generated
keytab using rc4-hmac.
Stopped samba
edit smb.conf and remove "use kerberos keytab = yes"
Deleted the existing computer account in AD
Deleted the existing mapped user account in AD
Deleted /etc/krb5.keytab
Edit krb5.conf and add rc4-hmac as -first- enctype in list for
default_tgs_enctypes, default_tkt_enctypes, permitted_enctypes
Deleted samba's private.tdb
Deleted samba's winbindd_cache.tdb (just in case)
Created a new windows user account to be used for mapping in ktpass.exe
Ran ktpass.exe on domain controller with "-DesOnly"
Read the new keytab and write /etc/krb5.conf with it

^^^^
Typo: should be /etc/krb5.keytab

Post by Doug VanLeuven
Run "net ads join"
Ethereal trace on port 88 show rc4-hmac negotiated tickets
Using a ktpass.exe generated keytab, the AD computer account and the
AD mapped user account attribute userAccountControl must agree on the
flag UF_USE_DES_KEY_ONLY. They either both indicate it or they
both don't indicate it, but they can't be mixed.
We'll be enjoying Thanksgiving holiday here.
Regards, Doug

[Christoph Kaegi]

Post by Doug VanLeuven
+des (des only - default for command)
-des (not des only)

I tried to create keytabs for this computer account
with all possible options -DESOnly, /crypto DES-CBC-CRC,
/crypto DES-CBC-MD5.

But as I always had "use kerberos keytab = yes", I guess
samba always overwrote the host/... principal anyway.

As this PAC thingy (I still don't know what it stands for)
doesn't seem to be important, I commented out the relevant
parts in smb_krb5_verify_checksum().
Surprise: The STATUS_LOGON_FAILUREs I mentionned in my
first mail still occurred.

Then It dawned on me, that I was possibly searching in
the wrong place.

After looking at the level 10 logs some more, I
found out, that samba was unhappy over the nonexisting
machine accounts of the clients.
(Local User Accounts here, synced by script)

Adding a machine account to my local /etc/passwd
seems to remedy the STATUS_LOGON_FAILUREs and
the corresponding delays.

Next step will be to either sync the AD machine accounts
to my local passwd also (which is sooooo ugly!)
or getting winbind with "idmap backend = idmap_ad"
to run, which I'm not too confident about...

Thanks for taking the time to help me, Doug and Guenther.

Chris

[Christoph Kaegi]

Post by Guenther Deschner

Post by Christoph Kaegi
-------------------------------------- 8< --------------------------------------
[2005/11/21 16:09:28, 3] libsmb/clikrb5.c:smb_krb5_verify_checksum(695)
smb_krb5_verify_checksum: krb5_c_verify_checksum() failed: Bad encryption type
[2005/11/21 16:09:28, 2] libads/authdata.c:check_pac_checksum(666)
check_pac_checksum: PAC Verification failed: Bad encryption type (-1765328196)
[2005/11/21 16:09:28, 0] libads/authdata.c:decode_pac_data(876)
decode_pac_data: failed to verify PAC server signature
[2005/11/21 16:09:28, 3] libads/kerberos_verify.c:ads_verify_ticket(416)
ads_verify_ticket: failed to decode PAC_DATA: NT_STATUS_ACCESS_DENIED
-------------------------------------- 8< --------------------------------------

Then you most probably are forced to use DES keys when authenticating with
Kerberos on your OS, right? PAC verification must then fail due to a bug
in Windows (which fails to put DES-based checksum into the PAC
signatures), so we can't verify the signature. What exact Kerberos library
are you using (version) ?

Today, I recreated the AD computer account. After issuing
the ktpass command on the domain controller, it said indeed:

"Account has been set for DES-only encryption"

Did I understand this correctly, that this is the desired behaviour?

Or should I specify -DESOnly?