Exclusive - ADP v Zenefits - a follow up and assessment
- Summary:
- We followed up with both ADP and Zenefits to get their take on events leading to the recent lawsuit. And to add our own take
The ADP and Zenefits spat is intriguing at multiple levels. We've already got the makings of an epic battle between the 800 lb. enterprise payroll gorilla and the wannabe small business benefits leader. While the opening shots were interesting, there were too many unanswered questions for us to do much more than speculate what is happening.
The basics are that Zenefits was accessing data from ADP's RUN payroll system on behalf of its customers — until ADP cut off that access and an almighty row broke out. We spoke with both sides to get a better understanding of what happened. Here is what they told us and our individual takes:
ADP
Phil, Den and Brian fielded a call with ADP executives. These included their Chief Security Officer and SVP of product development. The latter actually built the RUN platform as well as creating ADP's marketplace, which was launched last October to support an ecosystem of third-party application providers that can hook into a selection of its APIs.
ADP's RUN system is a payroll product targeted to smaller businesses with up to 50 or so employees. Currently, RUN has approximately 450,000 customers. Zenefits and ADP have around 1,000 mutual customers.
ADP executives told us that RUN customers often establish an administrator account so that a customer executive or third party (e.g., the customer's accountant) can access/update the information online. Apparently, most RUN customers and their advisors access their information via these administrator accounts one employee record at a time.
ADP systems (not just RUN) are accessed daily approximately 110,000 times to update the records of some 60 million customer employees. ADP stressed how diligently they monitor this access to ensure no unauthorized access or privacy hack is occurring. It is a result of this monitoring, they told us, that they noticed a spike in activity on the RUN system.
When the ADP security team investigated this matter, they came to the conclusion that the customer records being accessed had a common ingredient: they were joint ADP RUN/Zenefits customers.
ADP told us that third parties like Zenefits that want programmatic access to payroll data should go through their APIs. The first step is to become an authorized ADP partner and pay a partner fee of $2,500 to join the ADP Marketplace. They admitted however that RUN payroll data is not currently part of the Marketplace offering and so access to this via API would be subject to further negotiation "as part of a business development discussion."
The ADP executives we spoke to were adamant that Zenefits had made no approach at any time to discuss API access. They stated that their long-term goal is to have ADP payroll customers have access to all manner of partners (e.g., benefits providers, 401K providers, etc.). Their line is that they have spent millions creating secure connections to ADP systems and want their partners to follow the standards and tools they have created as part of this ecosystem.
When we pressed these executives on what happened, they speculated that something like a screen scraper program may have been accessing these RUN records. They encouraged us to hypothesize that if all 1000 customers were accessed simultaneously (or in continuous fashion) and each customer had 50 employees each with 20-30 screens of data with each screen containing possibly 50 items/data elements each, then several million I/O actions could occur in a short period of time. This was all hypothetical however: at no time did they explicitly reveal to us any details of exactly what did happen.
Regardless of the hypothesis, ADP claims it has the log files that prove the spike in activity that alerted them to what was happening. They claim it looked at first like a denial of service attack hence their shutting down access.
Zenefits
Due to time zoning constraints, we asked questions over email. Here is the verbatim text with responses:
Q: We understand that Zenefits used account administrator login credentials to access its customers' data programmatically from ADP RUN system (ie you wrote software that accessed information via an interface that ADP had designed for human users)
- Was this program essentially screen scraping data?
- Did the program access data from multiple customer accounts or records either simultaneously or in rapid sequence?
- What processes did you have in place to monitor this program's behaviour to make sure it did not overload ADP's systems or otherwise behave disruptively?
- Were there any occasions, either during testing or at other times, when this program may have generated an abnormal surge in activity on ADP's systems?
A: We have aggressive throttling protocols in place to make sure requests are spaced out and to limit multiple concurrent requests.
Looking at our activity, we did not see any spike in request volume to ADP. We shared this data with ADP, and asked to see what they saw on their end. They refused to share it with us, and said “you'll hear from our lawyers.” We asked to get our technical teams on the phone to discuss; they again told us to wait for their lawyers.
As far as how Zenefits interacts with ADP RUN, we are using the same exact interface as any other payroll administrator – just we’ve built software to automate time-consuming tasks.
Q: ADP say that you should have accessed this data through their APIs. We understand that this may have involved a charge and some custom development work as they do not currently make RUN payroll data available through their marketplace APIs.
- When did you first initiate discussions with ADP about API access to RUN data?
- What attempts did you make to negotiate with ADP for API access to the data you needed?
- What aspects of ADP's offer of API access were unacceptable to you?
A: First, the API didn't exist when we started building Zenefits, and if you check the ADP website, there is no API offered for RUN. Second, ADP shut us off on RUN; the API they do have is for Workforce.
Q: ADP has sued Zenefits for defamation, alleging that essentially you are milking this situation for market advantage.
- What was your motivation for going public with this so rapidly?
- What do you believe ADP should have done?
A: Let’s review what happened. We were getting calls from customers asking us what was going on as ADP was down the accounts they had created for Zenefits to access their payroll. We tried over and over to get ADP to engage with us on their alleged technical problems, they refused.
We owed it to our small business customers to explain what was happening, and what we could do to help them through this massive inconvenience ADP was causing to their operations. So, we emailed customers explaining our belief about what was going on.
Also remember that ADP was the first one to publish a public "statement" on their website -- a statement that questioned our security and spread a lot of false "FUD" about our system. We also were receiving media inquiries asking us to respond to what ADP had said. At that point, we decided that we had to respond.
We think that ADP should allow their engineers to talk to ours so we can resolve this, if they indeed believe the issue is a technical one. If there's some flaw they see in our system, we want to fix it. But they have refused to have that conversation with us. That, combined with the fact that they started deactivating accounts a week before the supposed "spike," and the fact that they started rolling out a service to compete with Zenefits' offering within days of taking this action, leads us to believe that the issue isn't technical but about restricting customer choice. Nevertheless, we are ready and willing to have a conversation with them to do whatever it takes to get this service turned back on for our mutual clients.
Our take
Brian Sommer
I'm intrigued by the interface program. In my career, I've seen perfectly functioning programs take down mainframes when a single bit of squirrelly data was encountered. A colleague recently told me of how a misplaced comma in the code of his program took out a supercomputer. He was banned from that data center by that tiny mistake. I tell you this as no one in this matter may have acted maliciously - the malicious acts could have been a byproduct of a faulty program or odd bit of data. I think it's important to keep this in mind as cooler heads may find that the real problem was technical not people. Right now, this case is a sea of emotion powered by an absence of data.
The litigation that's involved here, so far, is a defamation case but I suspect countersuits will soon fly. Lawyers will profit and mutual customers may be inconvenienced.
Meaningful communication between the two firms (nerd to nerd) may help get to the real root cause. Only then should the leaders of these firms get together and see if a solution for their mutual customers is possible.
Den Howlett
I am impressed that a company with ADP's history was willing to proactively reach out to us for additional comment. That almost never happens in litigious disputes of this kind.
I find it hard to believe that Zenefits actions on such a small sample of ADP total customer count could produce this level of activity such as to warrant a shutdown. I know Brian sees this differently but I would need to see compelling log file evidence to be satisfied that ADPs concerns were legitimate. Having said that, ADP's claims give weight to their rock solid security and performance monitoring, something that startups usually have to learn the hard way.
Zenefits is behaving like all Silicon Valley startups do when faced with an incumbent who is pushing back: beat the crap out of them using social media and see what sticks. The trouble with that is it only works when you are in the right and have public support. Right now, media has largely been behind Zenefits but that could change in a heartbeat.
I also question whether Zenefits knows what it's doing. Time and again we've seen startups build shoddy code only to find they really have little understanding about what it takes to run enterprise class software. If that's what's happened and I was an ADP security team member then I would have done exactly the same and shut them out.
I still think that ADP's defamation case is flimsy given the law in the state of California, the known public statements and the additional information we were able to glean but this mess needs resolving and quickly because right now it is customers who are caught in the crossfire.
As a side note, observers and analysts in the HR Tech Conference LinkedIn (invitation only) back channel have much to say on this topic. Taken in the round, I'd say their vote is with ADP at this stage.
Phil Wainewright
Zenefits, like many fast-growing Silicon Valley startups, is on a mission to change the world. It's busy beating off dozens of state regulators in a quest to upset the status quo in the health benefits market. It certainly hasn't got time to wade through whatever multi-month API access negotiations ADP would have imposed on it. Especially not if it can screen-scrape the data anyway by having its systems pose as a third-party administrator (and, its developers may have thought, who cares if ADP's systems grind to a halt, can't they look after themselves?)
ADP meanwhile has a 60+ year track record as a trusted guardian of corporate payrolls. It has a duty to ensure no one messes with those payrolls and it has no obligation to give a leg-up to an upstart competitor that is entering the market with an explicitly cavalier attitude to established practices. When its security staff discovered how Zenefits was accessing data I can imagine they must have been livid at the potential risk exposure.
What I don't understand is how this spat was allowed to escalate in a matter of hours into legal action. I don't think either party comes out of this well. ADP has allowed itself to be cast as hyper-sensitive and unwilling to compromise. Zenefits has been caught out playing fast and loose with a partner's systems without ever knocking on the front door to find out if there was a better way in.
These two vendors should put the lawyers to one side and start talking constructively. Both sides start off on the defensive, but ADP has most to lose here — Zenefits has already reaped a massive PR bonus in visibility and brand awareness. Meanwhile, small business customers of both vendors are the unwilling collateral casualties.
