Welcome to the SAMBA Domain: 10 years on

Posted 13 Feb 2009 at 21:22 UTC (updated 13 Feb 2009 at 21:59 UTC) by lkcl Share This

The 10th anniversary of "Welcome to the SAMBA Domain" quietly passed on Thursday 28th of August, 2007, without acknowledgement. When Paul Ashton and Luke Leighton initially reverse-engineered and published the NT Domains protocol, the floodgates were opened to both Free Software and Proprietary CIFS vendors to interoperate with Microsoft's flagship product, Windows NT, at an unprecedented level.

The anticipated reprisals from Microsoft did not happen: incredibly, instead, quiet mutual respect and cooperation crystallised the CIFS protocols into formal specifications (some of which were quietly handed out, whilst others had to wait until they were prised from Microsoft's fingers by the U.S. Dept of Justice and the E.U Commission).

In the intervening eleven years, an enormous amount has been achieved, yet, frustratingly, an enormous amount has not. This article outlines the accomplishments to date, and highlights the incredible things that could be achieved if some specific strategic inter-project free software cooperation took place. Also outlined are some hints as to how that can be accomplished, citing examples of prior proven work in which it has already been achieved, but not yet adopted.


SMB was introduced in fits and starts by numerous companies and organisations. It has a potted fragmented history that includes X-Open Specifications dating back to 1992; IBM providing an emulation layer of the underlying NetBIOS transport on top of TCP/IP, for both the UDP-equivalent (NetBIOS Datagrams - port 138) and the TCP-equivalent (NetBIOS streams - port 139). IBM had a further crack at SMB, with LANMAN 2.0; cross-licensing agreements between IBM and Microsoft resulted in that ending up in Windows and OS/2, and then Microsoft dropped IBM cold, with NT LANMAN 0.12 for Windows NT, with Windows NT set to become Microsoft's flagship operating system.

World-wide, Man-millenia of work has gone into reimplementing SMB, the transport layer and the services that run on top of it, by companies such as Microsoft, IBM, Network Appliances, UNISYS to name but a few, and groups such as the Samba and Samba TNG teams.

The reasons are simple: interoperability. There is no choice. If you want to interoperate in a Windows-dominated world; if you want to compile even the simplest of Win32 applications that will run seamlessly across many platforms, and in particular if you want the application to authenticate, the underlying infrastructure is forced to provide the authentication, transparently, for you, and that automatically draws in man-centuries of development effort, without your knowledge.

MSRPC was introduced in Windows NT 3.1 as part of the core infrastructure. The entire Operating System's critical services were written as MSRPC services. Key to this decision was the fact that Paul Leach, co-founder of Apollo, was the architect of NCA - Network Computing Architecture, and Apollo was one of the contributors to DCE/RPC. Freed from the constraints of The Open Group committee process, Microsoft was able to further enhance DCE/RPC to add critical necessary strategic enhancements, such as UCS-2 Unicode String support to NDR, and from there turn their eyes towards accidental world domination.

DCOM was announced in 1996 for Windows NT 4 and Windows 95, on top of prior releases of COM. Given that COM cannot really exist without the D, it's likely that the much despised Windows 95 team got their grubby little mitts on DCOM and, lacking any proper MSRPC infrastructure, simply ripped the D off the front. With the subsequent release of DCOM95.EXE, and the inclusion of DCOM in Windows NT 4.0 by default, the announcement of DCOM turned the computing world on its head, stifling competition and sucking developers further into the Win32 proprietary trap. Microsoft itself went on to make DCOM the basis of its major applications up until .NET attempted to take over. A quick scan through Wine's source code looking for "import stdole32.tlb" is a particularly enlightening exercise.

Interestingly, .NET itself is not linked, in any way, to DCOM (or MSRPC), which was clearly a strategy decision on Microsoft's part. Unfortunately, however, access to .NET services requires NTLM / NT authentication, and that is entirely implemented in MSRPC services.

.NET started out as a framework on top of a "Common Language Runtime". After the failure of efforts to push DCOM into the Unix and Linux worlds (with Software AG withdrawing their free DCOM product in 2004, and COMsource being available for a mere $USD 30,000), it can be surmised that Microsoft endeavoured to take a new tack. Visual BASIC, the only dynamically-typed scripting language that Microsoft had ever known, wasn't exactly taking the world by storm despite its seamless DCOM integration, and Python for Win32 which now includes DCOM support, was a long long way off.

So, Microsoft started again, with .NET, with a vision to provide support for many languages (even COBOL), and many platforms. The networked services use SOAP as the RPC mechanism, which can be dynamically-created and does not require a compiler such as MIDL.EXE - a distinct advantage. The specification for SOAP cross-references the DCE/RPC specification as inspiration for data types. SOAP is an abominally chatty and XML-based RPC mechanism that utilises HTTP as its transport. Authenticated HTTP on Microsoft's IIS Server utilises NTLMSSP. NTLMSSP utilises Microsoft's MSRPC-based NT Domain Authentication Services and Kerberos Authentication Services, and in this very indirect way it can be seen that .NET still depends on MSRPC, as do all key strategic Microsoft Technologies.

So, whilst it appears on the face of it, from looking at the MSDN library that .NET does not require MSRPC, thanks to authentication, full interoperability most definitely does require MSRPC and associated SMB and NetBIOS infrastructure (an additional man-century of coding effort: see "Samba Project Cost").

Some fifteen to twenty years later, Free Software Operating Systems are the only Operating Systems that have neither DCE/RPC nor DCOM at the heart of their infrastructure, or even any complete and self-consistent developer infrastructure or tool chains, in any form. Due to the lack of decent RPC and ORBs, attempts to recreate RPC and Inter-Process mechanisms and Object Request Brokers have resulted in D-Bus (a bastardised son of a whore for an IPC mechanism); KDE's home-grown inter-process communication mechanism DCOP (which is famously and proudly quoted as having been written in 20 minutes flat); Glib / Gobject (a highly successful but limited Object-Orientated interface, with startling similarities to COM, that cannot cross inter-process or inter-machine boundaries) and Gnome's "ORB", which uses CORBA, thanks to AT & T / Olivetti Lab's 1996 release of omniORB under the LGPL.

The strategic importance and power of DCOM and DCE/RPC is drastically and dramatically underestimated by the entire Free Software community. This article will outline the progress from the early beginnings - attempts made by Luke Leighton to address the balance - through to the current status of work achieved to date, starting right at the heart of the matter: Windows NT Authentication.

It will be seen that the goal was always to provide full network interoperability, but later on it became clear that Wine's infrastructure might even allow for developers to be seamlessly weaned off of dependence on the proprietary MSVC compilers, as well.

August 1997: "NT domain member to domain controller authentication protocol"

The "Green Light" for starting the NT Domains Research was Microsoft's release of NT 4.0 Beta. Within eight weeks, the critical first research was published in several forums, in a flurry of erroneous postings, by an over-keen and nervous Luke Leighton and an amused and laid-back Paul Ashton. Linus Nordberg followed up with a clean-room implementation in c of the published specification, which became the basis of Samba's NT Domain Primary Domain Controller functionality, and was re-published as crypt.txt

The significance of this breakthrough cannot be underestimated. Up until then, the only way to connect to a Windows NT Server was to use plain-text passwords! Windows NT 4.0 planned to disable plaintext passwords by default, which was a good idea, but Microsoft had absolutely no plans to reveal how to interoperate with the NT Domains infrastructure.

The design of NT Domains is conceptually similar to that of NIS+, Kerberos and DCE/DFS "Cells", all rolled into one, and it's no coincidence that later, in NT 5.0 (aka Windows 2000), both Kerberos and Directory Services were brought into the mix (in Microsoft's own inimicable way). Machines are securely joined to the Domain, and users can only log in from those machines (or from machines in a Domain where "Inter-Domain Trust" has been established).

The whole NT Domains architecture is based on top of DCE/RPC, which was subtly enhanced and augmented, for very good technical reasons such as adding support for UCS-2 Unicode strings, and called MSRPC. MSRPC and DCE/RPC are themselves the most incredibly powerful middleware technology that existed at the time, and are still arguably the best and also the most under-appreciated middleware technology in the world, having been used in projects such as running the distributed file server for the 1996 Olympics Web Site; and providing tens of thousands of distributed access points to the UK's 800 gigabytes (in the 1990s!) of National Insurance Database, and handling hundreds of thousands of queries per week (every single Employer in the United Kingdom).

At the time, the researchers into NT Domains were unaware of all this: they'd heard of DCE/RPC but believed it to be proprietary, despite a BSD-licensed freely available reference implementation of DCE/RPC 1.1 being ironically the basis of MSRPC.

A cursory examination of The Open Group's Technical Reference material led the researchers to conclude that TOG's morass of documentation was "so much horseshit", close examination of which would only slow them down in the process of network-reverse-engineering the MSRPC protocols. Thus, the future of the researchers was decided pretty much for the next two years: mind-numbing examination of literally thousands of network traces.

On reflection, it's difficult to determine whether ignoring TOG's detailed, comprehensive and amazing work was a good idea or not. Certainly, later, the DCE/RPC Technical Reference material proved absolutely invaluable, and the DCE 1.1 Reference Implementation turned up as FreeDCE some time later. Certainly, the floundering and meandering caused much hilarity at the number of security flaws and continual crashes of Microsoft's Flagship Operating System, many of which were reported, responsibly and confidentially over the ensuing months, to secure@microsoft.com with only a hint of cheekiness behind them.

It's questionable whether following the "proscribed" route, by using the DCE 1.1 Reference Implementation, would have resulted in so many security flaws detected, and, as a result, it's debatable as to whether the bun-fights inside Microsoft over this quotes blatant breach quotes of their quotes Intellectual Property Rights quotes would have come down on the side of "get rid of them, at all costs" instead of "leave them alone, they're useful".

After the initial contributions from Paul Ashton, Duncan Stansfield, Linus Nordberg and others, the fledgeling "Samba NTDOM" team began to take shape, with quiet contributions from Anders Blomdell, Luke Howard, Matthew Chapman, Jean-Francois Micouleau and others. Their chosen task was to map out the contents of Pandora's box that had been opened by the simple key of joining an NT Domain. Out the window was the "Old" IBM-inspired LANMAN protocols used by Windows 3.1 and Windows 95; In was approximately five hundred completely unknown functions, across well over twenty critical MSRPC services.

It would be six months before a tiny fraction of these were usefully implemented; two years before enough were implemented with confidence, and well over five years before Samba 3 achieved "stable" status with enough of these protocols to become a "mainstream player".

August 25th 1997: Microsoft announced extending DCOM beyond Windows

In a wonderful case of retrospective irony, a search on the Internet Archive for a TechNet article shows this: an announcement, three days before the "Welcome to the SAMBA Domain" announcement, that Microsoft had plans to extend DCOM onto Unix, Linux, Solaris and DEC platforms.

By 2004, Software AG had withdrawn their implementation of DCOM (which had been, unknowingly, freely available); COMSource, from The Open Group, was available for Solaris - full source code including the Microsoft MSRPC MIDL.EXE compiler and an implementation of a Windows Registry Service - for a mere $USD 30,000.

Coincidentally, by 2004, Wine had been forced to pretty much implement total binary and header-file interoperability with DCOM, anyway, leaving just the actual DCOM98.EXE service as the only remaining proprietary blob in the picture.

DCOM was the object-orientated wrapper on top of the "straight c" MSRPC layer, and it is through the use of the incredible technology that DCOM is that Microsoft was really able to utterly overwhelm what little competition there was. Only in retrospect is it possible to see how far advanced Microsoft's core strategic technology was ahead of everyone else, and Free Software, bless its little cotton socks, still really hasn't caught on, over ten years later. It didn't help that Olivetti Labs (bought by AT & T) released the first version of omniORB, in May 1997 (see AT & T Archive from 1999) because a Free Software release of CORBA resulted in CORBA becoming more dominant, and even adopted as part of Gnome's core infrastructure.

1998: NTLMSSPv2

By 1998, Luke Leighton was working for Internet Security Systems. Specific encryption schemes had been identified. It can be surmised that the NTLMSSPv2 protocol update was inspired by the initial network-reverse-engineering announcement of 1997, which showed to the world that the basis of Microsoft's Flagship Operating System, Windows NT, had an utterly borked 40-bit DES-based algorithm at its core. The only country to whom this really wasn't a problem was France, whose laws dictated that encryption, for authentication purposes, was illegal. (This turned out, later, to be extremely useful in reverse-engineering MSRPC services: install the "French" version of Windows NT from the MSDN and suddenly the previously encrypted MSRPC traffice was in-the-clear!).

After ninety days of research resulting in thirty lines of code, the workings of NTLMv2 and NTLMSSPv2 was established. It was quietly implemented, and quietly added to "The Alphabet Book", officially known as "DCE/RPC over SMB: Samba and NT Domains Internals".

On informing Microsoft privately of this news, a formal specification arrived by return, outlining the full details of the NTLMSSPv2 protocol. The subtext: NTLMv1 is so broken that it would be a really good idea if Samba had NTLMv2 added to it as soon as possible (unfortunately, main-line Samba was way behind Samba NTDOM and so it would be nearly four to five years later before Samba received full NTLMSSPv2 support in its MSRPC and SMB services, along with the NTLMSSPv2 128-bit NETLOGON Secure Channel).

Interestingly, protestations that the NTLMSSPv2 algorithm itself had a 64-bit bottleneck, despite its claimed 128-bit design, fell on deaf ears. Any such protestations were pretty irrelevant, anyway, as the design of NTLMSSPv2, like all of Microsoft's Core Technology enhancements, were a fait-accomplit development behind closed doors.

July 2000: Samba and Windows NT Security Interoperability

By July 2000, enough research had been done to provide Unix and NT Sysadmins with a high-level set of options where they could enable better security and interoperability between Samba and NT systems. This research was presented at Usenix's Lisa 2000 conference. A tool to suck the NT's SAM database over to a Samba Server were included. Hilarious gaffes in NT's SAM synchronisation which effectively exposed every single password of the entire NT Domain were a mere footnote, published as "Why You Should Upgrade To NT4 SP4 or NT5.

Also, a dig at the incredibly irresponsible attitude of key Microsoft Engineers is still, nearly a decade later, unheeded:

Hopefully, now that it is known that these systems are so insecure it will encourage the use of a more responsible and already-proven (time and time again) attitude to security. Namely that the only safe way to develop security measures is to publish their workings and let experts analyze it - before they are actually used in production environments.

The Alphabet Book had been published, with the choice of front cover image being particularly apt: a well-to-do gentleman helping himself to a well-off lady's pearl necklaces. It this book that many of the Proprietary CIFS vendors bought at least three copies, in the hope that it would help the author. The irony is that only 650 copies were actually sold before New Riders went out of business, and copies of the book are now available second-hand at 50 to 120% more than their retail value.

(side-note: can anyone explain why Amazon's sponsored links include "Refurbished DCE Unimaster" fumigation units?)

2000 - The Samba NTDOM Architecture grows legs

At some random point it became clear that to continue with a monolithic smbd process containing all of the MSRPC services would be crazy. The Samba TNG Man Pages outline the daemons implemented, with each MSRPC service implemented in its own application.

The TNG Daemon architecture brought interesting challenges, such as:

  • requiring NT security context information to be transferred securely between unix daemons running as root.
  • implementing an authenticated NamedPipe service, emulated on top of Unix Domain Sockets, including initially setting up the correct security context.
  • preserving MSRPC "threading" security contexts, which are when individual threads within an MSRPC application perform or require a different authenticated security context from the rest of the threads in the same client or server, over the same DCE/RPC channel!
  • getting smbd to not care about the data it is proxying, but acting as a go-between for the security context
  • providing a short-cut for all the vast amounts of authentication being performed when the transport can reasonably be identified as "local" - ncalrpc.
  • properly implementing NETLOGON SChannel which does recursive authentication in the case of Inter-Domain Trust.
  • Ensuring that smb.conf reloads worked as best as possible. The parameters were subdivided into three groups: connection, authentication and service, where connection comprised IP address, NetBIOS and DNS names; authentication introduced unix name and group, NT name and group macro substitution, and finally (the most contentious), connection to a service, which would always be IPC$ in the case of an MSRPC service, would result in the loss of continuity for smb.conf macro "includes".
  • Treating much of the infrastructure as if it were "joined to its own domain", by adding a machine account password for the machine itself. Even over loopback (on ncalrpc), this machine account would be utilised to securely authenticate the MSRPC services, thus preserving security, and also greatly simplifying the design of the TNG architecture by not forcing "loopback" communication down a separate special-case code path. Thus, a simple test "is the name dot or backslash backslash dot" if so use ncalrpc otherwise use ncacn_np would be all that was required to distinguish emulation of an NT workstation from emulation of a Primary Domain Controller!

Interestingly, it would be seven years before the Samba Team would begin to realise that they needed to implement a daemon architecture, and, despite rejecting the TNG architecture out-of-hand at the time, the Samba "Franky" architecture has the exact same design principles as Samba TNG, as it faces the exact same issues that were dealt with successfully, back in 2000.

In the intervening years, Samba would continue to dominate these critical ports and services, disallowing by design (as well as free software license incompatibility) any chance for other projects to contribute or interoperate.

A good analogy would be for the Linux Kernel developers to dictate to the Apache2 project that the Apache Server must be compiled and linked into the Linux kernel (not as a userspace service, but as a kernel-space driver), and, by inference, released as part of the Linux source code base and also, by being part of the Linux kernel, released under a GPL license.

Clearly that would be insane to expect, and would not in the least bit be tolerated if suggested, but that's exactly the situation, with regards to the NamedPipes transport, SMB IPC$ transport and the NetBIOS transports in Samba: Samba's monolithic design utterly dominates and excludes all options but those dictated by the Samba Team, and has done for over a decade.

2001: The Linuxcare Dot Com

By 2001, with the Dot Com bubble still flexing its muscles, Linuxcare had funding of $USD 100 million, and was looking to become the next Redhat or the next VALinux massive IPO gamble. Numerous big names such as Rusty Russell and Andrew Tridgell had been brought on-board, and it was on Andrew's recommendation that Luke Leighton was added to the Linuxcare Australia team, with a view to bringing main-line Samba finally up-to-speed with the research that was in the "NTDOM" branch.

The "NTDOM" branch was a fork of mainline Samba, back in 1998, and the fork point from which NTDOM took off was only after about eight months of research. As a result, Samba 2.0 had a very limited MSRPC implementation that remained stagnant for several years. This stagnation caused significant problems for Samba, and resulted in wasted contributions re-developing work that had already been successfully implemented and proven to work in NTDOM and TNG.

Linuxcare represented a badly-needed opportunity for Samba to "catch up".

November 2001: Winbind, the Single-Signon Holy Grail

The combination of Andrew's systems-level expertise and enthusiasm, Tim Potter's steady programming and Luke Leighton's long-standing research resulted in an outstanding achievement: Winbind. At last, it was possible to have a Unix system that could be part of a Windows NT Domain, with account names and account passwords stored in either a Microsoft proprietary Windows NT SAM Database, or even in a Samba Server's emulated SAM database! NSSwitch and PAM plugins merged NT Accounts and NT Passwords respectively into a Unix world, at last.

The advantages of using Samba and Winbind even in a pure Unix / Linux environment should be clear: tens or hundreds of thousands of accounts can be stored in a manageable secure database, subdivided by "Domains" if required, with Inter-Domain Trust established between departments, and, if necessary, and required, Windows NT Primary Domain Controllers brought into the mix.

With Samba and Winbind, even the Unix / Linux workstations are secured, as the workstations need to be added to the Samba-controlled Domain (in the same way that NIS+ operated).

The Linuxcare Bubble Bursts...

Linuxcare's treatment of Luke Leighton was almost criminally negligent. Internet Security Systems had been absolutely outstanding, with an H1-B1 visa obtained within months, and interim employment arranged with professionalism. So the precedent had been established, and the procedures worked out. These successfully-deployed procedures were relayed to Linuxcare's management team, with the expectation that they would be put into effect.

The warning signs had already emerged, when, even before joining, Linuxcare neglected to pay Luke Leighton's travel expenses to a CIFS Interoperability conference. Within three months, and outgoing costs of approximately £3500 per month with no pay cheques yet received or even discussed, Luke Leighton was 10,000 miles from home, £12,000 overdrawn, sitting in a hotel at 4am wondering what the **** was going on, and extremely distressed.

The effect that this situation had on relations with Andrew Tridgell were "not good", to put it mildly. It also didn't help that the entire Linuxcare Australia team, with the exception of Luke Leighton, were allowed to own their Intellectual Property, having all joined before the new Investors took control. Luke Leighton made a specific conscious commitment to terminate development of any source code and any further Samba-related research, until the Intellectual Property issue was resolved satistfactorily (which it wasn't). Fortunately, with over two years of research material to explain to Andrew, the decision not to develop further Intellectual Property was never going to an issue.

However, what was an issue was not the lack of new material, but the sheer quantity of old material. Along with the effects of the negligence of the Linuxcare management (which was eventually resolved - almost grudgingly and far too late), the transfer of knowledge to Andrew didn't go too well. It became clear that Andrew was both out of his depth, with respect to the comprehensive nature of the NT Domains protocols and the mismatch between the NT approach and Andrew's significant Unix systems-level knowledge, but also Andrew was becoming increasingly distressed by the effects that Linuxcare's negligence was having on Luke Leighton, as well.

By the time that Linuxcare began to address the issues (such as paying Luke Leighton regularly and adequately), the damage had long been done. Fortunately, the Winbind infrastructure is the lasting legacy of the Linuxcare fiasco: at least there was one positive contribution to the advancement of Free Software that came out of that awful time.

Insights, Losses and Responsibilities

Many years later, it can be seen that carrying out empirical observations to derive expert knowledge is a skill which leaves very little room for "exactitude and certainty". Offering "concrete proof" and "concrete evidence" is utterly impossible to do when the knowledge has been derived through literally thousands of manual, laborious test cases, many of which are long-since forgotten. When presenting such highly-derived information to people whose skills are specifically in the arena of Unix File Sharing, Unix systems programming, Unix Security, it can be seen that it would be really quite easy for such people to dismiss empirically-derived knowledge that appears to be so "etheric" and "flimsy".

It would also be incredibly easy to utilise expert Unix systems knowledge as a weapon to terminate arguments, and, when the subject being discussed is actually Windows NT Internal architecture, concrete and proven Unix knowledge clearly wins outright against complex conjecture and vastly stacked logical deduction.

In the end, then, it is users who have suffered most by this mismatch. More specifically, it is application developers who have been let down so badly, because users and Systems Administrators do not care about how MSRPC, DCOM. SMB and NetBIOS work, they care about how to use the services. In that regard, Samba continues to serve its users well; the Samba Team fulfil their implicit duty and their responsibilities to Free Software, and they deserve the utmost respect for doing so, for well over a decade.

Continuing with the documentation of advances in "Catching up" with Microsoft's NT Domains-based protocols, the subject of how developers are the ones that have been let down by the Samba Team will be discussed later, after careful review of further developments.

April 17th 2001 - Andrew Tridgell joins Jeremy Allison at VALinux

Samba Leaders join NAS Team

June 27th 2001 - VALinux drops NAS hardware (and the Samba Team)

ouch. The Dot Com bubble starts to implode. Jeremy Allison goes to work for HP; Andrew goes to work for Quantum (another NAS company).

Implications of HP's interest in Samba

Here's where something goes slightly awry with Samba's development. HP's interest in hiring Samba Team members is to ensure that the MSRPC spoolss service, and associated infrastructure, is working correctly (for the benefit of HP's Printing Division). With very little underlying infrastructure, the manually-created MSRPC layer puts a very large strain on the Samba infrastructure, and a considerable drain on the resources of the Samba team, for several years.

Jean-Francois Micouleau's initial work on spoolss for Samba NTDOM, had taken him nearly eighteen months to implement one of the biggest and most comprehensive of the NT Domains MSRPC Services: a whopping seventy functions. all by hand-coded marshalling (without an IDL compiler or even an IDL definition file - completely from on-the-wire network reverse-engineering). Jean-Francois' work on spoolss in Samba NTDOM alone expands the NTDOM source code by something like 30%. Even Wine's implementation of spoolss covers eleven DLLs.

At a crucial time, having strived so hard to ensure that control of Samba's development remained in their hands and their hands only, the Samba Team leaders were now being asked to incorporate infrastructure that they didn't fully understand. It would be nearly four years before this was achieved, even with a proven and successful implementation available to them, in the form of Samba TNG.

2002 - Luke Howard develops XAD: Active Directory for Unix

XAD was Luke Howard's implementation of Active Directory. Years before Samba's decision to reimplement Kerberos and LDAP, one person took FreeDCE, Heimdal, OpenLDAP and Samba, produced a few patches, implemented a PAULA Daemon, implemented proprietary SAMR, LSA and NETLOGON services (thanks to FreeDCE's BSD license), and released an interoperable Active Directory Server Product for Unix.

XAD Installation Instructions

It would be several years before Samba catches up with Luke Howard's outstanding achievements.

March 2003 - Samba 4 begins

Samba 4 begins development, with plans to introduce an MSRPC-interoperable MIDL compiler, and to reimplement much of Samba 3, all of MSRPC, all of the NT Domains MSRPC services, a DCOM server, and implementations of the Kerberos and LDAP protocols as well. (Note: Each of the free software projects that implement those protocols are at least a quarter of a million lines of code each, with multi-man-decades of specialist expertise and work gone into them, and the effort gone into the NT Domains Services measured in man-centuries of time spent).

No mention of interoperability for developers, at the Win32 level, is mentioned. At all. No mention of cooperation or interoperability for other projects (such as Wine) is made at this stage. Later on, Samba / Wine Conferences are arranged, as many people notice the similarity between the efforts of the Wine team and those of the Samba team.

The key focus of the Samba 4 development effort is on replacing the Samba 3 services, which are described as "showing their age". (Being based on code from 1998, at a time when very little was understood of MSRPC and its implications, that's an understatement of the first order).

25th May 2004 - Samba TNG ported to Win32 for use in ReactOS

In a Wine Interview with Steven Edwards, it's mentioned that Samba TNG has been ported to Win32, for deployment under ReactOS, the Free Software implementation of the Windows NT Operating System. This inspired Luke Leighton some time later to port FreeDCE and Samba TNG's hand-crafted MSRPC infrastructure to Win32, with a view to achieving the amusing goal of providing an entirely free software implementation of Microsoft's Flagship Operating System, as a Primary Domain Controller.

Aug 23rd 2004 - Advances in Samba 4

Advances in Samba 4 article is released. The most significant of the advances is the mention of the fledgeling reimplementation of MSRPC (despite the existence of a proven BSD-licensed reference implementation, FreeDCE). To date, Samba 4's pidl compiler still imposes a significant burden onto its developers, and leaves developers who would like to port Win32 DCOM or MSRPC applications to Free Software with such a significant maintenance nightmare on their hands.

In the paper, Volker also shows the significant age of the Samba 3 SAMR infrastructure, which was cut over from Samba NTDOM as far back as 1998, and remained pretty much stagnant ever since. In Samba TNG, the hand-marshalling of MSRPC was separated very early on from the actual "implementation":


ZERO_STRUCT(q_e); ZERO_STRUCT(r_e); if (!samr_io_q_del_groupmem("", &q_e, data, 0)) { return False; }

r_e.status = _samr_del_groupmem(&q_e.pol, q_e.rid);

return samr_io_r_del_groupmem("", &r_e, rdata, 0); }

RPC_IO_DECLARE(samr_io_q_del_groupmem, SAMR_Q_DEL_GROUPMEM, q_e) { RPC_MARSHALLER_INTRO(q_e);

RPC_MARSH_SUBCALL(smb_io_pol_hnd, q_e, pol); prs_uint32("rid", ps, depth, &q_e->rid));

return True; }

RPC_IO_DECLARE(samr_io_r_del_groupmem, SAMR_R_DEL_GROUPMEM, r_u) { RPC_MARSHALLER_INTRO(r_u);

prs_uint32("status", ps, depth, &r_u->status));

return True; }

These are the hand-marshalled equivalent of what MIDL.EXE and WIDL.EXE auto-generate from the IDL file, resulting in sam_s.c and sam_c.c which are the server-side marshalling and client-side marshalling, respectively.

This leave the "actual" implementation, _samr_del_groupmem(), in a separate file, and in Samba TNG there are no less than three separate SAM database daemons to choose from, thanks to this design: an LDAP samrd, an smbpasswd samrd and a TDB-based samrd. Here is the implementation of samr_del_groupmem in the "unix-like" samrd:

uint32 _samr_del_groupmem(const POLICY_HND * pol, uint32 rid)
    DOM_SID group_sid;
    uint32 group_rid;
    fstring group_sid_str;
    DOMAIN_GRP *grp;
        struct sam_passwd *sam_pass;

/* find the policy handle. open a policy on it. */ if (!get_policy_samr_sid(get_global_hnd_cache(), pol, &group_sid)) { return NT_STATUS_INVALID_HANDLE; }

sid_to_string(group_sid_str, &group_sid); sid_split_rid(&group_sid, &group_rid);

become_root(); sam_pass = getsam21pwrid(rid); unbecome_root();

if (! sam_pass) { return NT_STATUS_NO_SUCH_USER; }

DEBUG(10, ("sid is %s\n", group_sid_str));

if (!sid_equal(&group_sid, &global_sam_sid)) { return NT_STATUS_NO_SUCH_GROUP; } DEBUG(10, ("lookup on Domain SID\n"));

grp = getgrouprid(group_rid, NULL, NULL );

if (!grp) { return NT_STATUS_NO_SUCH_GROUP; }

if (!del_unix_groupmem(grp->name, sam_pass->nt_name)) { return NT_STATUS_ACCESS_DENIED; }

if (!del_group_member(group_rid, rid)) { return NT_STATUS_ACCESS_DENIED; }


The declaration of this function can clearly be seen to be completely compatible with the IDL file definition outlined in Volker's paper:

/* Function    0x18     */
NTSTATUS samr_DeleteGroupMember(
        [in,ref]                 policy_handle *handle,
        [in]                     uint32 rid

This is just one example, but it can be clearly seen that the plan for Samba TNG was always to migrate to an MSRPC interoperable compiler, as the entire suite of Samba TNG MSRPC services also have the hand-marshalled NDR infrastructure separated out from the actual implementation of the service, and the client-side hand-coded functions implemented as closely as possible to conform to the same IDL.

5th January 2005 - Andrew Bartlett's Samba 4 Active Directory Paper released

Samba 4 - Andrew Bartlett's Active Directory Thesis outlines how Samba 4 will provide Active Directory Server functionality. Also documented is the NTLM v1 and v2 authentication; how NTLMSSP is used in MSRPC and other areas; how Kerberos works and how Microsoft overloaded the PAC and much more.

It's interesting to note that, after Andrew Bartlet worked with Luke Leighton, with guidance from Luke Leighton on how NTLMSSP works in Samba TNG two years earlier, out of 64 references in the Thesis not one mentions Luke Leighton, in an area where he is an acknowledged expert. Not even in section in 10.1, which covers the History of Samba, is it mentioned that the Samba 2 DC functionality was from an early cut-over of Samba NTDOM work, from 1998. This in itself is quite an achievement. It's almost as if there was a desire to exorcise his existence from history.

7 Jan 2005 - Luke Leighton adds NT Authentication and ncacn_np (Named Pipes) to FreeDCE

A patch to FreeDCE is released which brings FreeDCE up-to-date and MSRPC on-wire interoperable. Integration with Samba TNG is added, hooking in to its NamedPipes emulation layer, allowing FreeDCE to have an "ncacn_np" transport. Support for the MSRPC "3-way-authentication" handshake is added to FreeDCE's authentication infrastructure.

20th Jan 2005 - Luke Leighton mentions NamedPipe interoperability on Wine lists

In a precursor to the Samba "Franky" architecture, Luke Leighton describes how ImpersonateNamedPipeClient could be implemented on top of a NamedPipes emulation layer.

In particular, the solution to "inheriting" security contexts over NamedPipes is described, by extending MSRPC to include a new PDU packet type. "Pre-Authentication" of a sorts, if you will, which of course can only be carried out over cast-iron-securable transports (such as root-only-accessible unix domain sockets).

By extending MSRPC in this way, it's possible for projects that use and implement DCE/RPC to interoperate in a "clean" manner, without having to worry about Free Software License incompatibilies. Any project that has an MSRPC-compatible IDL compiler can use the IDL file, containing the security context, as the specification.

No actual code need be exchanged or cross-licensed.

February 2005 - Luke Leighton Reverse-engineers MAPI and Exchange 5.5

Luke Leighton carries out MAPI and Exchange Exploration, sufficient to get Outlook to open an Inbox. IDL files are created for MAPI, emsabp and emsmdb. With the emsabp and emsmdb implemented in NDR, FreeDCE easily made mincemeat of them. MAPI, being a proprietary son-of-a-whore RPC format, required a different approach, and "aparser" - the precursor to pidl - was adapted and press-ganged into service.

An OpenChange developer picks up the developments, ports them to the current Samba 3 infrastructure, and, some time later, Samba Team members take notice and begin work on creating an OpenChange MAPI library.

Up until that exploratory work, no progress had been made on Exchange Interoperability: the OpenChange project had stalled, and had focussed on back-end JET Engine reverse-engineering. After the exploratory work, OpenChange is up-and-running, and is now a part of the Samba XP Conferences.

The primary focus of OpenChange is unfortunately on a MAPI client, not an Exchange Server replacement. Luke Leighton's and Matthew Chapman's early reverse-engineering accomplishments showed that progress is made much faster when the client, the server and the network tracing are all tackled at the same time: they each bootstrap each other forward and allow for better understanding and, crucially, a better test matrix framework.

22nd July 2005 - Wine DCOM success

Wine DCOM success.

Wine announces some progress in the implementation of DCOM interoperability. This interoperability is utterly necessary for key services and applications to work, in Wine. Missing from the Wine DCOM "picture" is the actual DCOM service (implemented in for example DCOM95.EXE and DCOM98.EXE). The rest of the Wine DCOM infrastructure is, incredibly, pretty much complete! Win32 DCOM Applications can be compiled as-is.

2nd February 2005 - Samba's DCOM progress

Implementing Distributed COM in Samba.

Jelmer outlines the progress made in implementing DCOM in Samba 4. Examination of the source code shows that it does not contain an implementation of DCOM "pinging", but it may contain enough for the Wine team to start replacing DCOM98.EXE.

As MSRPC binary-interoperability was never part of Samba 4's roadmap, let alone header-file interoperability, it's impossible for Win32 developers to utilise Samba 4's DCOM infrastructure for serious consideration of porting Win32 / DCOM applications to Unix.

It will be necessary to wait until the Samba "Franky" NamedPipes emulation layer is added to Wine, and also it will be necessary to wait until the Wine Team examine Jelmer's valuable efforts, and port them to Wine's much better MSRPC and DCOM infrastructure.

26 Jul 2005 - Samba TNG announces Embedded PowerPC success

Samba TNG announces a successful port to embedded systems. The internal daemon architecture allows non-essential MSRPC services to be completely removed with ease, dramatically reducing the size.

16 Sep 2005 - Samba TNG MSRPC Services successfully ported to Win32

An announcement is made indicating that rpcclient.exe is possible to compile for Win32, as is smbclient.exe, and also lsarpc.exe. binary downloads are made available, including nmbd.exe!

The msrpc-client.c commit log puts this as around 16th September 2005.

The significance of these endeavours is that ReactOS stands a chance to have SMB and MSRPC network interoperability. A native SMB Server for Win32 is essential: it's simply impractical to expect ReactOS to have "Services for Unix" and then run a unix-compiled Samba under that POSIX emulation layer, or worse, expect ReactOS to run the cygwin user-space POSIX emulation layer: the performance penalty and the known instability issues of cygwin would be atrocious.

20 Dec 2005 - Announcement of progress in porting FreeDCE to Win32

As part of providing ReactOS and Wine with a reference implementation of DCE/RPC (augmented to be MSRPC-compatible), FreeDCE was ported to Win32. Later, echoservice.exe and echoclient.exe were successfully created and announced.

Binaries available for download of rpcd.exe (the DCE/RPC Endpoint Mapper!), the pthread-win32 dll (pthreadGC2.dll), echoservice.exe and echoclient.exe.

April 2006 - Wine / Samba Conference highlights need for NamedPipes

Wine's Named Pipes Samba Interoperability Requirements Specification

Rob Shearer documents the discussions and notes from the Wine / Samba "cooperation meetings". No actual work is carried out until, two years later, Samba itself requires the exact same interoperability interface (yes - the exact same interface that had been successfully implemented in Samba TNG, seven years earlier).

2006 - Novell acquires PADL's XAD Technology (and announces it 2 years later)

Padl OpenEnterp Announcement.

Wonderful Novell. gotta love them.

2nd September 2007 - Linux MAPI

Dan Shearer and Julien Kerihuel present Linux MAPI Programming over ExchangeRPC at the 2007 LinuxConf. The significant progress on integration of the MAPI client into numerous Free Software GUI front-ends is described, along with the limited progress on the Free Software Exchange Server replacement. Overall it's an outstanding achievement, at last.

2nd September 2007 - Wine Samba Networking Interoperability

Kai Blin's Wine Samba Integration talk describes some of the options for integrating Wine and Samba. This is the early precursor to the "Franky" Design.

D-Bus is mentioned in this early talk as one of the options. (Note: D-Bus is a bastardised version of all the best features of DCE/RPC shrivelled up into a pathetic attempt at inter-process communication. A comparison of the D-Bus specification and The Open Group DCE/RPC specification shows direct concept-for-concept equivalence, with the unfortunate fact being that the D-Bus specification only bothers to cover about 20% of the ground that DCE/RPC covers. Of particular concern is the lack of an IDL compiler in D-Bus, which is one of the absolutely vital features of any IPC mechanism, without which you might as well simply use 1,500 lines of c-code to transfer data between processes using unix domain sockets. Conclusion: D-Bus should have been shot at birth).

Sep 6th 2007 - Samba 4 Active Directory Interoperability announced

A Samba 4 AD article explains the status of Samba 4's Alpha 1 Active Directory Server interoperability release, with a warning that it is in no way ready for production use.

May 8th 2008 - Samba "Franky" Design

Samba Franky a combination of the best parts of Samba 3 and Samba 4. The architecture, utilising Unix Domain Sockets, is identical, concept-for-concept, to the Samba TNG Daemons architecture that had been implemented nine years previously.

It's ironic that internal interoperability requirements finally forced the Samba Team to make the right decision that will truly "Open Windows to a Wider World", not the Samba TNG proven architecture, nor the discussions in November 2004, nor the requests by the Wine Team to open up the protocols, in 2006.

April 17th 2008 - OpenChange Server progress

OpenChange Samba XP 2008 talk outlines the progress made with OpenChange Exchange Interoperability. The MAPI Client library is good: the OpenChange Server, utilising Samba 4 infrastructure, is less functional.

Dec 2008 - Novel announces Active Directory Interoperability

Press Release

After purchasing XAD from Luke Howard nearly two years previously, and three years after entering into a License with the Devil that cost them one of their key strategic assets (Jeremy Allison), Novell announces that they have Active Directory Interoperability.

30th Jan 2009 - Kai Blin mentions Samba "Franky" Design on Wine-Devel

In a NamedPipes thread, Kai Blin mentions that Samba "Franky" at long last has the beginnings of the infrastructure that will make it possible for other developers and other projects to interoperate and communicate, thanks to a "Named Pipes emulation layer" inside forked versions of Samba 3 and Samba 4.

Later in the thread, Luke Leighton mentions how pleased he is that finally Samba looks like it's actually going to deliver on its promise of "Opening Windows to a Wider World", and offers some guidance from his experience nine years previously in this exact same specific technical area.

5th Feb 2009 - Luke Leighton ports "Com in Plain C" to winegcc

As an illustration to developers of how easy it is to develop DCOM applications utilising an entirely Free Software toolchain, Luke Leighton ported COM in Plain C, a well-respected guide to writing DCOM applications in c, to Wine.

The purpose of illustrating how easy and simple this is was to highlight just how far Free Software, and Wine, has come, and to illustrate exactly why the Samba Team has been holding things up with their intransigence over implementing NamedPipes.


Looking back on the developments since that initial "Welcome to the SAMBA Domain" announcement is fascinating. Due to lack of funding, and thanks to Luke Leighton's reputation as a leading Free Software developer having been destroyed, specific and limited strategic contributions are instead made which have a profound, indirect, hated, despised, entirely unwelcome but ultimately and eventually really grudgingly - and utterly privately - accepted influence on the direction of the Samba Team.

The most successful of these quiet strategic contributions has to be the MAPI exploration. Thanks to that unknown developer's reimplementation, the Samba Team had no idea as to where the work had come from, and so ran with it, with enthusiasm. The reaction of one of the Samba Team members, on discovering in 2007 the provenance of the work they were doing, was really quite violent.

So it took many years for Samba 3 to catch up to where Samba TNG had already successfully gone, with Samba TNG regularly being examined as a reference guide to understand NTLMSSP, NTLMv2, the NETLOGON SChannel, the internal workings of the key strategic MSRPC Services such as SAMR, NETLOGON and LSARPC, and much more.

The benefits of the internal architecture of TNG as separate daemons for each MSRPC service was clear as daylight, with the benefits and possibility for swapping out individual services on a case-by-case basis and replacing them with, for example, Wine-derived services with a real purpose behind them, being blindingly obvious. Wine already has three really important services that could clearly bring significant benefits to Free Software if turned into proper networked MSRPC services: SVCCTL (for starting and stopping services); SPOOLSS (for printing) and WINREG (the Wine Registry Service).

Even the Samba "Franky" design will benefit enormously from such an architecture, as the best components from all of the projects in this field - FreeDCE itself, Wine, Samba 3, Samba 4 and Samba TNG - can be picked not by the Samba Team but by any developer or systems administrator. Inter-project cooperation can only be a good thing; encouraging cross-project testing across such a well-designed and well-defined interface such as DCE/RPC will only result in significant improvements to each (as the CIFS Interops already prove).

The addition of Kerberos and LDAP into the mix is almost a minor hiccup - a footnote or a speedbump taken at 60mph - and as Luke Howard's patches showed, once you have SAMR, NETLOGON and LSARPC client and server infrastructure, extending Heimdal and OpenLDAP and then adding PAULAD is pretty straightforward. Even the undocumented PAC format isn't hard, as it must comprise everything that a NET_USER_INFO_3 structure already contains.

The lack of support for DCOM is a significant failure, even on my part, for which I can only whole-heartedly apologise to all the people who have been let down, by having to pay extortionate Microsoft Licensing fees, and for having no choice but to utilise proprietary compiler technology and libraries. Unfortunately, this is what happens when you don't financially reward the right people - the ones who quietly get on with the job: they have to take up work as common labourers, or earn money doing web sites, instead of helping break the back of a billion-dollar monopoly to give you back your freedom.

I have one really significant apology to make, and it's this: that I did not have the strength at the time to overcome the egoistic bullying that I was subjected to. The consequences alone of Samba not having a NamedPipes emulation layer are beyond measure, financially, but run into billions of dollars of your money, forcibly spent on proprietary technology because there was no alternative. I made an attempt in November 2004 to re-engage with the Samba Team, to impress on them the absolute paramount strategic importance of a NamedPipes emulation layer. After a simple acknowledgement from Jeremy Allison that it was a good idea, I let you all down by getting very distressed at the amount of time that had elapsed since i was last forced into silence when previously discussing the same technical issue, having been informed in no uncertain terms that it was a useless idea that would never work and would never be securely implemented, especially by someone who had been told in clear unequivocable terms was technically incompetent at Unix systems programming.

Well, once again, Samba TNG is proven to be the correct technical approach. Nine years late, Samba "Franky" has been forced to implement a NamedPipes emulation layer, utilising unix domain sockets, transferring NT Security Contexts over them, and providing the full semantics of NT's NamedPipes. Quite literally exactly as I had implemented, in 2000.

Once again, the Samba Team has been shown the way, by an unpaid, unfunded and much disrespected Free Software advocate who keeps coming back, despite people's belief that he is insufferably arrogant. The definition of arrogance is a combination of ignorance and ego. Where I know I am not knowledgeable, far from wishing to hide my ignorance behind an egoistic mask, I am overwhelmingly happy to be corrected (willing to learn so that I do not waste any further time in achieving my goal), willing to learn with eagerness and alarming speed. The rate at which I incorporate new knowledge has always been one of the things that has made it so difficult for people to accept me - especially those people who lack confidence, and my willingness to admit ignorance and be corrected was - and often still is - perceived as "weakness". And regarding ego: if I am egoistic, then why am I exposing myself to the continuous ridicule by re-raising these issues year on year? Surely, a truly egoistical individual, bent solely on scoring a few points against those much-respected leaders of a significant free software project, would have given up a long, long time ago?

No, the fact is this, that even after a decade of silence, I still have a detailed understanding of this scarily-large technological topic spanning millions of lines of code and literally man-centuries of development effort. I have a clear and unequivocal appreciation of how strategically important these protocols are, in ways that not even those who have presented evidence to the E.U Commission and the U.S. Dept of Justice fully appreciate. Consequently, I have a duty and a responsibility wayyy beyond that of most people that you will ever encounter in your life, to ensure that those people who have pushed me aside for their own personal reasons and their own personal gain, and thus have taken over in part my responsibilities to you, are hounded until their absolute last dying breath to ensure that they actually fulfil the responsibilities that they have assumed.

And those responsibilities do not just include delivery of File and Print Services for Unix, as is believed to be so important: that responsibility has been achieved a long time ago, and the Samba Team deserve our utmost respect and praise for delivering and maintaining such services. No - there is much more: also included is responsibilities and a duty to bring to developers the freedom from proprietary toolchains such as MSVC and the Microsoft Developer Network, in this little-understood, obscure, esoteric and incredibly powerful technical area. It is in this area that both I and the Samba Team are letting you down so badly, for which i genuinely, genuinely apologise.

At the top of the chain of key strategic Microsoft protocols is DCOM, with pretty much everything underneath that single protocol spanning millions of lines of code and dozens of different services. When it is possible for an entirely unencumbered free software toolchain to compile and run a DCOM application - better yet, when it is possible to compile even Python-DCOM with an entirely unencumbered free software toolchain, running under an entirely free software infrastructure on the platform of your choice, and for that application to interoperate seamlessly with other Win32-based DCOM applications, then my responsibility and duty to you will have been fulfilled.

I trust and hope that that will be well before 2019.

Lastly, I have to reiterate that my praise of the Samba Team for finally beginning the implementation of a NamedPipes architecture is truly genuine, because a NamedPipes layer truly is the way to "Open Windows to a Wider World".

Family feud, posted 14 Feb 2009 at 06:08 UTC by cdfrey » (Journeyer)

Thanks for the article. It was an interesting read.

Sadly, it sounds like some of the road blocks along the way were due to a bit of a private feud that may still be going on today. And the result turns into individual castles of development.

And with human nature being the way it is, I'm sure that articles like this won't help the feud at all. It may point some people toward alternate technical realities, but the feud itself may remain.

These kinds of feuds and competitions are often hidden completely from the average user or admin. They don't know, and often don't care. They just want it to work. Indeed, how many admins actually know and understand all the existing features of Samba, let alone that a Samba TNG exists and what it can do for them? There is just too much to learn, and the average admin's installation choices are what keep Samba on top of the heap.

The average employer also doesn't know about such feuds, but does care. News of feuds like this smell like competition, or uncertainty, and nobody wants to invest in something shaky or uncertain or old fashioned. People want to invest in progress when it benefits their own interests, but sometimes those same people don't know their own interests until they are explained to them.

You write about developers that quietly get the job done, meaning you, but this article is not quiet. I'm not saying this to discourage you from writing it. In fact, I was really wishing there was an article like this so I could hear the history and the feud presented from both sides. But I'm a developer, and very interested in technology, and try to expand my knowledge in order to do my job better. Competition and tech battles between experts gives bystanders an opportunity to evaluate the issues for themselves, and hopefully come away richer, even if the original experts remain deadlocked.

As I gain experience in the free software world, I'm learning that sometimes two people, or two projects, can have strongly held beliefs as to the definition of the Right Way (TM). It can be frustrating, for users and for the developers involved, to see multiple independent threads of effort going in similar directions. It can easily be the case that both sides are convinced that their architecture is the right one, both with perfectly valid reasons. Critiques of those architectures may only help to solidify their views, depending on the person.

This kind of thing is as old as vi vs. emacs, Gnome vs. KDE, and Linux vs. BSD.

It takes a "big man" to avoid this duplication, and this personality feature is not always found in the geek crowd. And sometimes, especially in the case of funded projects, it is not a decision one person can make.

What is my solution to all this? I think it can only be fixed with education, which has a chance, albeit slim, to change the general direction of people's minds. Not just developers' minds, but users' and admins' and employers' minds as well.

And for a technology as staggeringly massive as Samba, Wine, MSRPC, DCE/RPC, and FreeDCE, a metric crap-ton of education is needed.

For example, why in the world should the average developer care that D-Bus is a "bastardized son-of-a-whore" when it is already installed on his system, has a potentially simpler API, and gets his job done? This is an honest question, because I'm one of the developers who would like to know. :-)

The only way I'll understand this is if I take the time to learn FreeDCE, and that's hard work. And this is from a developer who's actually peeked at the code a little. I'm sure there are lots who just haven't heard of FreeDCE and the various versions available and where to get them, etc.

Even after reading this article, while I'm nebulously excited at the existing potential, I'm not confident in my understanding of what is truly possible in the Samba and MSRPC worlds today, even if I were to venture outside the stock Samba feature set; and definitely not in terms that I could easily explain to a fellow admin or pitch to a venture capitalist.

It is this kind of clarity that people need. They need that lightbulb to go on in their heads, so that even people well-versed in standard Samba could open their eyes wider and ask "I can actually do that today?"

If clear articles like that were available, I'm sure they would be snapped up by magazines like Linux Journal and friends. But such articles need to be focused on the present and the future, instead of the past.

For example, when Luke Howard released his XAD, were there any magazine articles on it at the time? My rudimentary searches don't find any.

And before I close, it occurs to me that people who are good at their jobs may not always be good at selling themselves (me included), and that this discrepancy often gets worse the better they are at their jobs. If top actors, pro athletes, script writers, and book authors can benefit from the services of an agent, perhaps top developers can too. Does this already exist and I just haven't heard of it?

My pie-in-the-sky wishlist in conclusion: :-)

  • a multi-part series of articles on advogato on how to use FreeDCE
  • more articles in Linux Journal and on the web about all the pieces related to Windows compatibility, including Samba, Wine, DCE/RPC, FreeDCE, and the compiler toolchain
  • good free software developers landing good jobs in their specialty, using agents if needed

thank you, posted 14 Feb 2009 at 14:27 UTC by lkcl » (Master)

chris, thank you - your reply raises several key questions and is very insightful, especially on short notice.

one of the reasons why i was "right" so-to-speak is because the rather odd combination of implementing NT Domains infrastructure on top of POSIX-compliant systems can really only be done in an extraordinarily limited number of ways. for example, to communicate securely and efficiently between isolated processes, _the_ only way is to use unix domain sockets. TCP/IP sockets certainly cannot be made to be secure; a socketpair can only be created inside a single process, and passing the fd across to another isolated process is non-portable and requires a Unix Domain Socket to do it, anyway! Shared memory could potentially be located, by an extensive search.

So the only portable way is unix domain sockets. but, the problem you face is that some unixen will not allow you to create a unix domain socket with file permissions set (e.g. solaris), so you have to create a subdirectory and set permissions on that:

int create_pipe_socket(const char *dir, int dir_perms,
               const char *path, int path_perms)
#ifndef HAVE_OS_WIN32
    int s;
    struct sockaddr_un sa;

DEBUG(4, ("create_pipe_socket: %s 0%o %s 0%o\n", dir, dir_perms, path, path_perms));

sys_mkdir(dir, dir_perms);

if (sys_lstat(dir, &sbuf) != 0 || !S_ISDIR(sbuf.st_mode)) { DEBUG(0, ("Error setting up pipe dir %s\n", path)); return -1; }

if (chmod(dir, dir_perms) < 0) { DEBUG(0, ("chmod on %s failed\n", dir)); return -1; }

if (remove(path)!=0 && errno != ENOENT) { DEBUG(0, ("remove on %s failed\n", path)); }

/* start listening on unix socket */ s = socket(AF_UNIX, SOCK_STREAM, 0);

if (s < 0) { DEBUG(0, ("socket open failed\n")); return -1; }

ZERO_STRUCT(sa); sa.sun_family = AF_UNIX; safe_strcpy(sa.sun_path, path, sizeof(sa.sun_path));

if (bind(s, (struct sockaddr *)&sa, sizeof(sa)) < 0) { DEBUG(0, ("socket bind to %s failed\n", sa.sun_path)); close(s); remove(path); return -1; }

if (s == -1) { DEBUG(0, ("bind failed\n")); remove(path); return -1; }

if (path_perms != 0) { chmod(path, path_perms); }

if (listen(s, 5) == -1) { DEBUG(0, ("listen failed\n")); return -1; }

DEBUG(5, ("unix socket opened: %s\n", path));

return s; #else return -1; #endif }

Through using such unix-systems-level techniques, I was informed by Andrew in clear and precise terms that I was incompetent and unfit to do such work. but he didn't actually even _look_ at what I had done. So, for winbindd, he had Tim Potter reimplement the inter-process communication system - using unix domain sockets:

/* Create winbindd socket */

static int create_sock(void) { struct sockaddr_un sunaddr; struct stat st; int sock; mode_t old_umask; pstring path; /* Create the socket directory or reuse the existing one */

if (lstat(WINBINDD_SOCKET_DIR, &st) == -1) {

if (errno == ENOENT) {

/* Create directory */

if (mkdir(WINBINDD_SOCKET_DIR, 0755) == -1) { DEBUG(0, ("error creating socket directory %s: %s\n", WINBINDD_SOCKET_DIR, strerror(errno))); return -1; }

} else {

DEBUG(0, ("lstat failed on socket directory %s: %s\n", WINBINDD_SOCKET_DIR, strerror(errno))); return -1; }

} else {

/* Check ownership and permission on existing directory */

if (!S_ISDIR(st.st_mode)) { DEBUG(0, ("socket directory %s isn't a directory\n", WINBINDD_SOCKET_DIR)); return -1; } if ((st.st_uid != 0) || ((st.st_mode & 0777) != 0755)) { DEBUG(0, ("invalid permissions on socket directory %s\n", WINBINDD_SOCKET_DIR)); return -1; } } /* Create the socket file */ old_umask = umask(0);

sock = socket(AF_UNIX, SOCK_STREAM, 0);

if (sock == -1) { perror("socket"); return -1; }

snprintf(path, sizeof(path), "%s/%s", WINBINDD_SOCKET_DIR, WINBINDD_SOCKET_NAME);

unlink(path); ZERO_OBJ(sunaddr); sunaddr.sun_family = AF_UNIX; safe_strcpy(sunaddr.sun_path, path, sizeof(sunaddr.sun_path));

if (bind(sock, (struct sockaddr *)&sunaddr, sizeof(sunaddr)) == -1) { DEBUG(0, ("bind failed on winbind socket %s: %s\n", path, strerror(errno))); close(sock); return -1; }

if (listen(sock, 5) == -1) { DEBUG(0, ("listen failed on winbind socket %s: %s\n", path, strerror(errno))); close(sock); return -1; }


/* Success! */

return sock; }

As you can see, the similarities between the code that Andrew didn't even look at, and the code that he got Tim to write which he then accepted, is startling (Tim's face said it all when I showed him the two pieces of code, at the time).

Ten years later I can still quote five or six strategically vital incidents like this, and every single one of them was key to providing the world with a much more "open" free software implementation of these protocols.

Every single one was dismissed with whatever argument could be thought up: in one case, Andrew's voice was near to pleading with me to accept the words that even he could hear, coming out of his own mouth, sounded wrong. I still remember his face - his voice almost trembling and raising in query at the end of the sentence, eyebrows raised, to encourage me to work _with_ him to concede the "argument" and allow him to be "right".

You have to wonder what the hell was going on, in the mind of such an incredibly respected, talented and skilled individual, on whose shoulders there is so much responsibility. And - as you point out - so little is actually _known_ about those responsibilities he is entrusted with. And you wonder why it was that both he and Jeremy chose to use their knowledge and expertise, instead of for the betterment of free software, instead to "make themselves look right".

I never told Jeremy, certainly not at a dinner paid for by one of the CIFS conference sponsors, when he told me that he was "right all along", and "how wrong I had been", and how delighted he was that Winbind had a proper implementation of mapping Unix uids/gids onto NT SIDs - the words I did not speak at the table were, "Jeremy, you xxxxing idiot, I *designed* Winbind. Whenever Tim was stuck and could see that Andrew didn't understand what was needed, knowing that Andrew would interfere with whatever argument he could come up with, Tim would wait until he wasn't around and ask *me* how it should *really* be done". Instead I walked away, too upset to speak, and returned later so as not to embarrass our hosts.

Whenever I raise these issues to point out to such incredibly highly regarded, respected, awarded and rewarded individuals that they are not fulfilling their duty, I am made out to be a criticiser, a whiner, a loser, an incompetent programmer, someone who "wanted to be right" - you name it, it's used against me to smash me down and dismiss what I'm saying - _anything_ but actually thinking, "christ, this guy's been on these people's backs for ten years: why isn't he giving up, is there actually something _to_ what he's been going on about, and should we be asking the people that we're entrusting with the enormous responsibility to help end Microsoft's monopoly some _really_ pointed questions?"

Because that's really what this is about - ending Microsoft's accidental monopoly, both for "users" as well as "developers".

And any egos, any "I have to be right whatever the cost", all the bullying has got to stop. There's only a limited number of ways in which this stuff can be done, and it's dictated not by the design of POSIX / Unix / Linux but by the design of Windows NT.

You think I added the TNG daemons architecture - the exact same architecture that's being implemented in Samba "Franky" - so i could show samba team members i was "right"?? I did it because it was the only option to fulfil very specific technical criteria.

Retrospectively, the Samba Team members saw year after year of reimplementing my work that I had logically followed a chain of breadcrumbs a xxxxing marathon long, as they were forced to follow in the exact same footsteps that I had been forced to follow. They let you down _so_ badly by going "NO! WE ARE RIGHT! YOU ARE NOT FIT TO DO THIS WORK!", and I let you down by letting them get away with doing that.

which is why i apologised so much, at the end of the article.

Still in the past, posted 15 Feb 2009 at 21:51 UTC by cdfrey » (Journeyer)

I hear what you're saying, but what else could you have done? Given the distribution of resources, was there a different tactic that would have produced different results in the past 10 years?

If not, then you shouldn't be so hard on yourself, or the Samba team. There would be nothing for you to apologise for, since there was nothing else that could be done. And being hard on the Samba team won't change the last 10 years, and probably won't effect any new change in the next 10.

But if there was a different tactic that would have produced different results, then there's no use crying over 10 years of spilled milk (although documenting it somewhere is probably a good idea), but instead that new tactic should be put into use today.

One lesson that I find really hard to learn, personally, is that as human beings, failure is often the path to success. Sometimes it is the only path to success.

And damn, that takes a long time. :-)

None of us are the same people we were 10 years ago, and the lessons we've learned over the last 10 years make us capable of more today, but that level of capability could not have been expected of us during that 10 years of learning.

It appears obvious to me that 10 years ago, you had the technical aspects solved way ahead of the pack. On the other hand, other people involved had the financial and political aspects solved ahead of you.

Which is kinda why I brought up the idea of an agent in my last post. There are people that just have no interest or knack for the political side of things. To some, it is disgusting that there even exists a political side. But that doesn't make politics go away, and those that succeed either learn to work the system for themselves, or unite with those that can.

So in the case of Samba, it seems to me that the following things are needed:

  1. a list of key missing features, clearly explained and documented for the masses
  2. a list of what is possible, today, using all Free Software, not just Samba.. again, for the masses
  3. someone to work on the key missing features
  4. someone to pay that person to work on the key missing features
  5. consistent, non-threatening publicity on the progress of those features

None of this requires the Samba team in order to work. But the Samba community and support structure can be used to help things along.

It seems that #4 and #5 are holding things back.

Item #4 is where politics comes in. That's where evangelism and advocating what's possible comes in. That's where hope and joy and change and bright futures and clarity and user-friendliness come in. This is not a place to dwell on any injustices of the past.

Item #4 is also where shrewd thinking comes in. It is where nothing is given away for free. It is where deals are struck, between equals on the negotiating table. It is where nobody, ever, spends $3,500 per month and goes $12,000 into debt to make it happen, and where only those with the money on hand take the risks. It is where work stops if the money stops, and as soon as the money stops.

Item #4 also means that none of this will get done at any significant rate of speed if #4 doesn't exist. And nobody should feel guilty about that. If you are the expert in this field, and you only have time and funds to make tiny free contributions to Samba in your free time, then the world loses out, but it's not your fault.

As for #5, Samba has mailing lists, and uses git! This is perfect. The person who works on missing feature X can just fork his own repository, and make regular announcements of his success to the mailing list. No pressure to get it included into Samba. No pressure for the Samba team to change direction. That programmer only needs to keep going full-blast in his own direction, pay the utmost attention to any users that show up and ask questions, and let the community apply the pressure back on the Samba team. The less pressure that comes from the developer, and the more that comes from the community, and the more it looks like the developer is working in the best interests of Samba and the Free Software community, the less resistence there will be to incorporating his work.

Everyone involved needs to be able to save face.

It looks like you already did #5 to some extent. Samba TNG wouldn't have happened otherwise. It's too bad that there wasn't enough #4 to make a complete successful fork at the time.

I just now took a peek at the FAQ on samba-tng.org, and I must say, it doesn't sound very confident. If I was a random admin trying to decide whether to use Samba or Samba TNG, that FAQ would not convince me to switch to Samba TNG. It might even convince me to stick with Samba and forget about TNG completely!

How many admins read that page and made the same decision?

And if that was the case, it's no wonder there was little community pressure for Samba to change course. That page is a wiki, though, so perhaps it was worded more strongly in the past.

Anyway, that's how this case appears to me. Maybe I'll see it differently in 10 years after I've learned more about life. :-) But that's the best I can do right now.

bar stewards, posted 15 Feb 2009 at 21:55 UTC by lkcl » (Master)

For example, why in the world should the average developer care that D-Bus is a "bastardized son-of-a-whore" when it is already installed on his system, has a potentially simpler API, and gets his job done? This is an honest question, because I'm one of the developers who would like to know. :-)


why should the average _user_ care that DCOP was written in 20 mins (and, as a developer, you can tell...) when it's already installed, has an API that's so simple it's insane, and gets the job done?

let's see, shall we?

what does D-Bus offer? a means to connect processes together and transfer data. you get nothing more than what unix domain sockets, an endpoint mapper and some hton and ntoh (byte-ordering) macros provide.

what does DCOP offer? a means to connect processes together and exchange simple-structured data. strings, integers and lists are supported. there's an IDL compiler, allowing you to make functions and for the functions to even have parameters.

what does DCE/RPC offer? christ, where do i start. DCE/RPC provides as you already know, chris :)

  • a means to connect processes together, irrespective of what machine they're on.
  • a means to connect applications together across networks, irrespective of the word-order of the two machines.
  • a rich set of basic data types, ranging from ASCII and EBDIC through 8 to 64-bit integers up to IEEE standardised floating point arithmetic of _different_ types.
  • a rich set of complex data types of _variable_ sizes
  • support for pointers to structures including but not limited to support for transferring doubly-linked lists for god's sake i mean - do the designers of XMLRPC, JSONRPC and all the other simple RPC mechanism sleep well at night, knowing that their "best shot" has been completely blown away by the programming.... _gods_ who developed DCE/RPC.
  • an incredible IDL compiler that puts all of these data structures together, grouping them into functions. called "interfaces".
  • version control on interfaces i mean who thought up _that_ one, it's pure genius. made a mess of your design but it's too late, it's already released? no worries - just add extra functions and the "old" applications will still be able to operate whilst you upgrade first the servers and then the clients, and the DCE/RPC infrastructure will help the distributed applications negotiate down to whatever level of interface is common to both client and server. absolute genius.
  • security negotiation
  • per-thread security contexts.

      i could go on, but you get the point. and, you could say "oh weeell, it's alll tooo complicated, we couldn't _possibly_ need that, ouuur lovely system is sooo simple it woooorks, look at how little code our system is, it's only N,000 lines of code, while that DCE/RPC crap is a quarter of a million" but then you look a little closer at the projects, using the "simple" RPC mechanism, and you find that the projects are multi-man-decade projects with one hundred developers, twenty million lines of code and _severely_ creaking at the seams.

      the KDE guys _really_ wanted to drop DCOP but didn't quiiite have the guts to take on DCE/RPC, which is a _real_ pity, because it means that they would have got things like being able to run the "front-end" of an app on one machine and have the actual "back-end" run on another.

      remote administration "for free".

      the people i described this to said "oh no, that's crap - all you need to is ssh into the box and edit the kde config files" oh purleeeze :)

      the next person said, "oh no, that's crap - all you need is to run X-Windows over ssh tunnelling, or use NoMachine" oh purleeze, have you _tried_ running X-windows over even a high-speed broadband, it's _shit_, and have you tried setting up NoMachine, it's a _bitch_ to get right.

      so - the opportunity is lost, with KDE 4 eating the D-Bus dogfood, and having to struggle with no support for anything but that which they manually marshal and unmarshal for themselves.

      ... good luck :)

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

Share this page