Diffy

Status

Diffy is used in production at:

• Mixpanel
• Airbnb (Scalabity) (Migration)
• Twitter
• Baidu
• Bytedance

and blogged about by cloud infrastructure providers like:

• Microsoft
• Google
• Alibaba Cloud
• KrakenD
• Datawire

Visit our engineering blog to keep up with the latest developments in Diffy.

Diffy is being actively developed and maintained by the engineering team at Sn126.

Feel free to contact us via discord, linkedin, or twitter.

What is Diffy?

Diffy finds regressions in your service by running your new code and your old code side by side and comparing what they return. It behaves as a proxy that multicasts every request it receives to each running instance, compares the responses, and renders a verdict — Safe to ship when the candidate matches the primary within noise tolerance, or Regressions detected when real differences surface.

The premise is simple: if two implementations of a service return “similar” responses for a sufficiently large and diverse set of requests, the newer implementation can be treated as regression-free.

How does Diffy work?

Diffy acts as a proxy that accepts requests drawn from any source you provide and multicasts each of those requests to three different service instances:

1. A candidate instance running your new code
2. A primary instance running your last known-good code
3. A secondary instance running the same known-good code as the primary

When those services respond, Diffy compares the responses and looks for two things:

1. Real differences observed between the candidate and the primary.
2. Non-deterministic noise observed between the primary and the secondary. Since both run known-good code, you should expect them to agree. Where they don’t, your service is exhibiting non-deterministic behavior — Diffy treats that signal as noise.

Diffy measures how often primary and secondary disagree with each other vs. how often primary and candidate disagree. If those rates are comparable, Diffy concludes the candidate has no real regression and the difference is just noise — which is what the signal-to-noise indicator in the UI summarizes at a glance.

The Diffy UI

Diffy ships with a single-page UI organized around five views:

• Overview — the run verdict (Safe to ship / Regressions detected), signal-to-noise, traffic and diff timeseries, and the top failing endpoints.
• Endpoints — drill into any endpoint, walk the field tree, and open side-by-side or three-way diffs in the inspector.
• Noise — review which fields are flagged as noise and tune cancellation rules per endpoint.
• Transformations — define request/response transformations applied before comparison.
• Runs — browse the history of past comparison runs and switch between them.

Documentation

Detailed Diffy Documentation is available here.

Support

Please reach out to [email protected] for support. We look forward to hearing from you.

Code of Conduct

1. Bug reports are welcome even if submitted anonymously via fresh github accounts.
2. Anonymous feature and support requests will be ignored.

License

Copyright (C) 2019 Sn126, Inc.

This license allows reusers to copy and distribute the material in 
any medium or format in unadapted form only, for noncommercial purposes 
only, and only so long as attribution is given to the creator. 

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Public License
for more details.

You should have received a copy of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0
International Public License along with this program. If not, see 
https://creativecommons.org/licenses/by-nc-nd/4.0/.